You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Sven Herzky <sv...@mail.rakuten.com> on 2014/03/27 12:36:20 UTC

Reducing compaction queue more efficiently

Hey all,

I've put some data (~ 2.5 TB) into an HBase table on a small cluster (8 
dn/rs + 1 master, max region size 10 GB, having ~ 350 regions) and 
collected something around 1500 entries in my compaction queue (~ 200 
per regionserver). Unfortunately, HBase now is slow when pushing new 
data into it and it just compacts with 0.02 per second (getting also 
"WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow)" in log files).

If I want to trigger the compaction manually, there's nothing happening:

hbase(main):001:0> compact 'images'
0 row(s) in 1.3690 seconds

hbase(main):002:0> major_compact 'images'
0 row(s) in 2.3670 seconds

I'm currently using cdh4 with HBase 0.94.

Do somebody have a clue how to speed up the process or why the manual 
compaction doesn't work as "expected" (by me)?

Thanks in advance,
Sven

Re: Reducing compaction queue more efficiently

Posted by Sven Herzky <sv...@mail.rakuten.com>.
Hey,

Thank you for your reply. Wasn't aware of the async behaviour 
(output/help of hbase shell is a little bit miss-leading in this aspect).

Is there a way to speed the process up or "ensure" immetiately working 
on that?

Regards
Sven

On 27.03.2014 15:52, Ted Yu wrote:
>  From src/main/ruby/hbase/admin.rb :
>
>      def major_compact(table_or_region_name, family = nil)
>        if family == nil
>          @admin.majorCompact(table_or_region_name)
>        else
>          # We are major compacting a column family within a region or table.
>          @admin.majorCompact(table_or_region_name, family)
>        end
>      end
>
> In HBaseAdmin:
>
>     * Major compact a table or an individual region.
>
>     * Asynchronous operation.
>
>     *
>
>     * @param tableNameOrRegionName table or region to major compact
>
>     * @throws IOException if a remote or network exception occurs
>
>     * @throws InterruptedException
>
>     */
>
>    public void majorCompact(final String tableNameOrRegionName)
>
>
> Cheers
>
>
> On Thu, Mar 27, 2014 at 4:36 AM, Sven Herzky
> <sv...@mail.rakuten.com>wrote:
>
>> Hey all,
>>
>> I've put some data (~ 2.5 TB) into an HBase table on a small cluster (8
>> dn/rs + 1 master, max region size 10 GB, having ~ 350 regions) and
>> collected something around 1500 entries in my compaction queue (~ 200 per
>> regionserver). Unfortunately, HBase now is slow when pushing new data into
>> it and it just compacts with 0.02 per second (getting also "WARN
>> org.apache.hadoop.ipc.HBaseServer: (operationTooSlow)" in log files).
>>
>> If I want to trigger the compaction manually, there's nothing happening:
>>
>> hbase(main):001:0> compact 'images'
>> 0 row(s) in 1.3690 seconds
>>
>> hbase(main):002:0> major_compact 'images'
>> 0 row(s) in 2.3670 seconds
>>
>> I'm currently using cdh4 with HBase 0.94.
>>
>> Do somebody have a clue how to speed up the process or why the manual
>> compaction doesn't work as "expected" (by me)?
>>
>> Thanks in advance,
>> Sven

Re: Reducing compaction queue more efficiently

Posted by Ted Yu <yu...@gmail.com>.
>From src/main/ruby/hbase/admin.rb :

    def major_compact(table_or_region_name, family = nil)
      if family == nil
        @admin.majorCompact(table_or_region_name)
      else
        # We are major compacting a column family within a region or table.
        @admin.majorCompact(table_or_region_name, family)
      end
    end

In HBaseAdmin:

   * Major compact a table or an individual region.

   * Asynchronous operation.

   *

   * @param tableNameOrRegionName table or region to major compact

   * @throws IOException if a remote or network exception occurs

   * @throws InterruptedException

   */

  public void majorCompact(final String tableNameOrRegionName)


Cheers


On Thu, Mar 27, 2014 at 4:36 AM, Sven Herzky
<sv...@mail.rakuten.com>wrote:

> Hey all,
>
> I've put some data (~ 2.5 TB) into an HBase table on a small cluster (8
> dn/rs + 1 master, max region size 10 GB, having ~ 350 regions) and
> collected something around 1500 entries in my compaction queue (~ 200 per
> regionserver). Unfortunately, HBase now is slow when pushing new data into
> it and it just compacts with 0.02 per second (getting also "WARN
> org.apache.hadoop.ipc.HBaseServer: (operationTooSlow)" in log files).
>
> If I want to trigger the compaction manually, there's nothing happening:
>
> hbase(main):001:0> compact 'images'
> 0 row(s) in 1.3690 seconds
>
> hbase(main):002:0> major_compact 'images'
> 0 row(s) in 2.3670 seconds
>
> I'm currently using cdh4 with HBase 0.94.
>
> Do somebody have a clue how to speed up the process or why the manual
> compaction doesn't work as "expected" (by me)?
>
> Thanks in advance,
> Sven
>