You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Xu-Feng Mao <m9...@gmail.com> on 2011/09/06 10:15:52 UTC
big compaction queue size
Hi,
We're running a 33-regionserver hbase cluster on top of cdh3u0 suites. On
average, we have 2400 regions hosted
on each regionserver. (hbase.hregion.max.filesize is 1.5GB, and we have
value size up to 4MB per object).
I check the log of regionserver, it seems like the compaction queue size is
about 1700, and every the compaction action
takes about 1 minute, and more over, most of the compaction are triggered to
a major one.
My question are,
1. Would this cause the performance degradation? It seems like "GET" action
in the interval that two minutes before/after
the compaction takes much longer time than usual. I thought the compaction
is a asynchronous operation.
2. Any issue would cause long-term compaction?
3. It seems like HBASE-1476 is going to implement multi-threaded compaction,
I guess it would help to reduce
the size of compaction queue.
Thanks and regards,
Mao Xu-Feng
RE: big compaction queue size
Posted by Zhoushuaifeng <zh...@huawei.com>.
In hbase version 0.90.x, compact running only one thread, so if your input is too fast, compact will be not able to finish in time.
This 3 parameters may help:
hbase.hstore.compactionThreshold default 3, increase it, e.g. 10
hbase.hstore.blockingStoreFiles e.g. 30
hbase.hstore.compaction.max e.g. 20
How many is suitable depends on the store file size flushed by regionserver, if it's too small, you can increase the count more.
Even more, you may need to control the input speed.
Zhou Shuaifeng(Frank)
-----Original Message-----
From: Xu-Feng Mao [mailto:m9suns@gmail.com]
Sent: Tuesday, September 06, 2011 4:16 PM
To: hbase-user@hadoop.apache.org; user@hbase.apache.org
Subject: big compaction queue size
Hi,
We're running a 33-regionserver hbase cluster on top of cdh3u0 suites. On
average, we have 2400 regions hosted
on each regionserver. (hbase.hregion.max.filesize is 1.5GB, and we have
value size up to 4MB per object).
I check the log of regionserver, it seems like the compaction queue size is
about 1700, and every the compaction action
takes about 1 minute, and more over, most of the compaction are triggered to
a major one.
My question are,
1. Would this cause the performance degradation? It seems like "GET" action
in the interval that two minutes before/after
the compaction takes much longer time than usual. I thought the compaction
is a asynchronous operation.
2. Any issue would cause long-term compaction?
3. It seems like HBASE-1476 is going to implement multi-threaded compaction,
I guess it would help to reduce
the size of compaction queue.
Thanks and regards,
Mao Xu-Feng
Re: big compaction queue size
Posted by Gaojinchao <ga...@huawei.com>.
Do you check your cluster has flushed many little hfiles? How much memory is used ?
It seems that region server has too many active regions
You can use this command check it:
grep "Finished memstore flush" *regionserver* -c
grep "global heap pressure" *regionserver* -c
-----邮件原件-----
发件人: Xu-Feng Mao [mailto:m9suns@gmail.com]
发送时间: 2011年9月6日 16:16
收件人: hbase-user@hadoop.apache.org; user@hbase.apache.org
主题: big compaction queue size
Hi,
We're running a 33-regionserver hbase cluster on top of cdh3u0 suites. On
average, we have 2400 regions hosted
on each regionserver. (hbase.hregion.max.filesize is 1.5GB, and we have
value size up to 4MB per object).
I check the log of regionserver, it seems like the compaction queue size is
about 1700, and every the compaction action
takes about 1 minute, and more over, most of the compaction are triggered to
a major one.
My question are,
1. Would this cause the performance degradation? It seems like "GET" action
in the interval that two minutes before/after
the compaction takes much longer time than usual. I thought the compaction
is a asynchronous operation.
2. Any issue would cause long-term compaction?
3. It seems like HBASE-1476 is going to implement multi-threaded compaction,
I guess it would help to reduce
the size of compaction queue.
Thanks and regards,
Mao Xu-Feng
Re: big compaction queue size
Posted by Gaojinchao <ga...@huawei.com>.
Hi J-D
Should we can give a formula about active regions per node and up to book ? I think many people encounter the same problem.
I think the formula is:
If( (Hlognumber*hdfsblock) > (HBASE_HEAPSIZE *memstore.lowerLimit) )
Active Regions = (HBASE_HEAPSIZE *memstore.lowerLimit )/( flush.size / (2~3))
Else
Active Regions = (Hlognumber*hdfsblock)/ (flush.size / (2~3))
If I am wrong, please correct. Thanks.
-----邮件原件-----
发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
发送时间: 2011年9月7日 1:19
收件人: user@hbase.apache.org
主题: Re: big compaction queue size
Inline.
J-D
> We're running a 33-regionserver hbase cluster on top of cdh3u0 suites. On
> average, we have 2400 regions hosted
> on each regionserver. (hbase.hregion.max.filesize is 1.5GB, and we have
> value size up to 4MB per object).
2400 region is just too many, if you are importing data at a high rate
(which might be the case with such fat values) and well distributed
among those regions then you will be force flushing tons of small
files everytime a region server needs to clean a log file.
Try to set it to 100 per node, then disable splitting, and also set a
bigger flush size on your table.
>
> I check the log of regionserver, it seems like the compaction queue size is
> about 1700, and every the compaction action
> takes about 1 minute, and more over, most of the compaction are triggered to
> a major one.
>
> My question are,
>
> 1. Would this cause the performance degradation? It seems like "GET" action
> in the interval that two minutes before/after
> the compaction takes much longer time than usual. I thought the compaction
> is a asynchronous operation.
It's async, but still uses IO resources which may impact latency.
Compacting also creates new blocks so the block cache is churning
through invalid blocks.
> 2. Any issue would cause long-term compaction?
It's more like a systemic issue, everything influences everything else.
> 3. It seems like HBASE-1476 is going to implement multi-threaded compaction,
> I guess it would help to reduce
> the size of compaction queue.
But also generate more IO, in your case 1476 would probably not help at all.
Re: big compaction queue size
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Inline.
J-D
> We're running a 33-regionserver hbase cluster on top of cdh3u0 suites. On
> average, we have 2400 regions hosted
> on each regionserver. (hbase.hregion.max.filesize is 1.5GB, and we have
> value size up to 4MB per object).
2400 region is just too many, if you are importing data at a high rate
(which might be the case with such fat values) and well distributed
among those regions then you will be force flushing tons of small
files everytime a region server needs to clean a log file.
Try to set it to 100 per node, then disable splitting, and also set a
bigger flush size on your table.
>
> I check the log of regionserver, it seems like the compaction queue size is
> about 1700, and every the compaction action
> takes about 1 minute, and more over, most of the compaction are triggered to
> a major one.
>
> My question are,
>
> 1. Would this cause the performance degradation? It seems like "GET" action
> in the interval that two minutes before/after
> the compaction takes much longer time than usual. I thought the compaction
> is a asynchronous operation.
It's async, but still uses IO resources which may impact latency.
Compacting also creates new blocks so the block cache is churning
through invalid blocks.
> 2. Any issue would cause long-term compaction?
It's more like a systemic issue, everything influences everything else.
> 3. It seems like HBASE-1476 is going to implement multi-threaded compaction,
> I guess it would help to reduce
> the size of compaction queue.
But also generate more IO, in your case 1476 would probably not help at all.