You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Sven Herzky <sv...@mail.rakuten.com> on 2014/03/27 12:36:20 UTC
Reducing compaction queue more efficiently
Hey all,
I've put some data (~ 2.5 TB) into an HBase table on a small cluster (8
dn/rs + 1 master, max region size 10 GB, having ~ 350 regions) and
collected something around 1500 entries in my compaction queue (~ 200
per regionserver). Unfortunately, HBase now is slow when pushing new
data into it and it just compacts with 0.02 per second (getting also
"WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow)" in log files).
If I want to trigger the compaction manually, there's nothing happening:
hbase(main):001:0> compact 'images'
0 row(s) in 1.3690 seconds
hbase(main):002:0> major_compact 'images'
0 row(s) in 2.3670 seconds
I'm currently using cdh4 with HBase 0.94.
Do somebody have a clue how to speed up the process or why the manual
compaction doesn't work as "expected" (by me)?
Thanks in advance,
Sven
Re: Reducing compaction queue more efficiently
Posted by Sven Herzky <sv...@mail.rakuten.com>.
Hey,
Thank you for your reply. Wasn't aware of the async behaviour
(output/help of hbase shell is a little bit miss-leading in this aspect).
Is there a way to speed the process up or "ensure" immetiately working
on that?
Regards
Sven
On 27.03.2014 15:52, Ted Yu wrote:
> From src/main/ruby/hbase/admin.rb :
>
> def major_compact(table_or_region_name, family = nil)
> if family == nil
> @admin.majorCompact(table_or_region_name)
> else
> # We are major compacting a column family within a region or table.
> @admin.majorCompact(table_or_region_name, family)
> end
> end
>
> In HBaseAdmin:
>
> * Major compact a table or an individual region.
>
> * Asynchronous operation.
>
> *
>
> * @param tableNameOrRegionName table or region to major compact
>
> * @throws IOException if a remote or network exception occurs
>
> * @throws InterruptedException
>
> */
>
> public void majorCompact(final String tableNameOrRegionName)
>
>
> Cheers
>
>
> On Thu, Mar 27, 2014 at 4:36 AM, Sven Herzky
> <sv...@mail.rakuten.com>wrote:
>
>> Hey all,
>>
>> I've put some data (~ 2.5 TB) into an HBase table on a small cluster (8
>> dn/rs + 1 master, max region size 10 GB, having ~ 350 regions) and
>> collected something around 1500 entries in my compaction queue (~ 200 per
>> regionserver). Unfortunately, HBase now is slow when pushing new data into
>> it and it just compacts with 0.02 per second (getting also "WARN
>> org.apache.hadoop.ipc.HBaseServer: (operationTooSlow)" in log files).
>>
>> If I want to trigger the compaction manually, there's nothing happening:
>>
>> hbase(main):001:0> compact 'images'
>> 0 row(s) in 1.3690 seconds
>>
>> hbase(main):002:0> major_compact 'images'
>> 0 row(s) in 2.3670 seconds
>>
>> I'm currently using cdh4 with HBase 0.94.
>>
>> Do somebody have a clue how to speed up the process or why the manual
>> compaction doesn't work as "expected" (by me)?
>>
>> Thanks in advance,
>> Sven
Re: Reducing compaction queue more efficiently
Posted by Ted Yu <yu...@gmail.com>.
>From src/main/ruby/hbase/admin.rb :
def major_compact(table_or_region_name, family = nil)
if family == nil
@admin.majorCompact(table_or_region_name)
else
# We are major compacting a column family within a region or table.
@admin.majorCompact(table_or_region_name, family)
end
end
In HBaseAdmin:
* Major compact a table or an individual region.
* Asynchronous operation.
*
* @param tableNameOrRegionName table or region to major compact
* @throws IOException if a remote or network exception occurs
* @throws InterruptedException
*/
public void majorCompact(final String tableNameOrRegionName)
Cheers
On Thu, Mar 27, 2014 at 4:36 AM, Sven Herzky
<sv...@mail.rakuten.com>wrote:
> Hey all,
>
> I've put some data (~ 2.5 TB) into an HBase table on a small cluster (8
> dn/rs + 1 master, max region size 10 GB, having ~ 350 regions) and
> collected something around 1500 entries in my compaction queue (~ 200 per
> regionserver). Unfortunately, HBase now is slow when pushing new data into
> it and it just compacts with 0.02 per second (getting also "WARN
> org.apache.hadoop.ipc.HBaseServer: (operationTooSlow)" in log files).
>
> If I want to trigger the compaction manually, there's nothing happening:
>
> hbase(main):001:0> compact 'images'
> 0 row(s) in 1.3690 seconds
>
> hbase(main):002:0> major_compact 'images'
> 0 row(s) in 2.3670 seconds
>
> I'm currently using cdh4 with HBase 0.94.
>
> Do somebody have a clue how to speed up the process or why the manual
> compaction doesn't work as "expected" (by me)?
>
> Thanks in advance,
> Sven
>