You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Upender Nimbekar <ni...@gmail.com> on 2014/01/14 16:21:14 UTC
Rebuild HBASE Table to reduce Regions per RS
Hi,
Does anyone have any experience rebuidling the HBASE table to reduce the
number of regions. I am currently dealing with a situation where the no. of
regions per RS have gone up quite significantly (500 per RS) and thereby
causing some performance issues. This is how I am thinking of bringing it
down:
increase the hbase.hregion.max.filesize from 500 MB to 2 GB
And the rebuild the HBASE table. I am assuming if after table rebuild, I
should see the no. of regions come down to more than half. I would
basically like to stay within HBASE suggested no. of regions per RS which
is about 50-200.
Please suggest if someone has any experience doing it.
Thanks
Upen
Re: Rebuild HBASE Table to reduce Regions per RS
Posted by Dhaval Shah <pr...@yahoo.co.in>.
If you can afford downtime for your table, there are ways to do it. You can:
- Merge regions (requires table to be disabled atleast in some older versions and probably in newer ones too)
- Go brute force by doing an export, truncate, import (this is a little more manageable when you have a large number of regions). This however is way more resource intensive and will take longer.
If you can't afford downtime, I would suggest create another table which mirrors this one and then switch to the new one
Regards,
Dhaval
________________________________
From: Upender Nimbekar <ni...@gmail.com>
To: dev@hbase.apache.org; user@hbase.apache.org
Sent: Tuesday, 14 January 2014 10:21 AM
Subject: Rebuild HBASE Table to reduce Regions per RS
Hi,
Does anyone have any experience rebuidling the HBASE table to reduce the
number of regions. I am currently dealing with a situation where the no. of
regions per RS have gone up quite significantly (500 per RS) and thereby
causing some performance issues. This is how I am thinking of bringing it
down:
increase the hbase.hregion.max.filesize from 500 MB to 2 GB
And the rebuild the HBASE table. I am assuming if after table rebuild, I
should see the no. of regions come down to more than half. I would
basically like to stay within HBASE suggested no. of regions per RS which
is about 50-200.
Please suggest if someone has any experience doing it.
Thanks
Upen
Re: Rebuild HBASE Table to reduce Regions per RS
Posted by Ted Yu <yu...@gmail.com>.
HBASE-7403 provided online merge capability.
The usage is:
HBaseAdmin#mergeRegions
Note: online merge is in 0.96 and above releases.
On Tue, Jan 14, 2014 at 11:00 AM, Vladimir Rodionov
<vl...@gmail.com>wrote:
> Nice. Ted. Is there any reason why we can't do it online?
>
>
> On Tue, Jan 14, 2014 at 10:47 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Upender:
> > For 15.2.2 Merge, please note the following condition:
> >
> > LOG.info("Verifying that HBase is not running...");
> > try {
> > HBaseAdmin.checkHBaseAvailable(getConf());
> > LOG.fatal("HBase cluster must be off-line.");
> >
> > Cheers
> >
> >
> > On Tue, Jan 14, 2014 at 10:40 AM, Vladimir Rodionov
> > <vl...@gmail.com>wrote:
> >
> > > I have never tried this before but I think the following should work:
> > >
> > > 1. Alter your table:
> > >
> > > habse> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '5000000000'
> > > (place your own number here)
> > >
> > > 2. Merge regions:
> > >
> > > http://hbase.apache.org/book/ops.regionmgt.html
> > >
> > >
> > >
> > >
> > > On Tue, Jan 14, 2014 at 7:21 AM, Upender Nimbekar <
> > > nimbekar.upender@gmail.com> wrote:
> > >
> > > > Hi,
> > > > Does anyone have any experience rebuidling the HBASE table to reduce
> > the
> > > > number of regions. I am currently dealing with a situation where the
> > no.
> > > of
> > > > regions per RS have gone up quite significantly (500 per RS) and
> > thereby
> > > > causing some performance issues. This is how I am thinking of
> bringing
> > it
> > > > down:
> > > >
> > > > increase the hbase.hregion.max.filesize from 500 MB to 2 GB
> > > >
> > > > And the rebuild the HBASE table. I am assuming if after table
> rebuild,
> > I
> > > > should see the no. of regions come down to more than half. I would
> > > > basically like to stay within HBASE suggested no. of regions per RS
> > which
> > > > is about 50-200.
> > > >
> > > > Please suggest if someone has any experience doing it.
> > > >
> > > > Thanks
> > > > Upen
> > > >
> > >
> >
>
Re: Rebuild HBASE Table to reduce Regions per RS
Posted by Ted Yu <yu...@gmail.com>.
HBASE-7403 provided online merge capability.
The usage is:
HBaseAdmin#mergeRegions
Note: online merge is in 0.96 and above releases.
On Tue, Jan 14, 2014 at 11:00 AM, Vladimir Rodionov
<vl...@gmail.com>wrote:
> Nice. Ted. Is there any reason why we can't do it online?
>
>
> On Tue, Jan 14, 2014 at 10:47 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Upender:
> > For 15.2.2 Merge, please note the following condition:
> >
> > LOG.info("Verifying that HBase is not running...");
> > try {
> > HBaseAdmin.checkHBaseAvailable(getConf());
> > LOG.fatal("HBase cluster must be off-line.");
> >
> > Cheers
> >
> >
> > On Tue, Jan 14, 2014 at 10:40 AM, Vladimir Rodionov
> > <vl...@gmail.com>wrote:
> >
> > > I have never tried this before but I think the following should work:
> > >
> > > 1. Alter your table:
> > >
> > > habse> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '5000000000'
> > > (place your own number here)
> > >
> > > 2. Merge regions:
> > >
> > > http://hbase.apache.org/book/ops.regionmgt.html
> > >
> > >
> > >
> > >
> > > On Tue, Jan 14, 2014 at 7:21 AM, Upender Nimbekar <
> > > nimbekar.upender@gmail.com> wrote:
> > >
> > > > Hi,
> > > > Does anyone have any experience rebuidling the HBASE table to reduce
> > the
> > > > number of regions. I am currently dealing with a situation where the
> > no.
> > > of
> > > > regions per RS have gone up quite significantly (500 per RS) and
> > thereby
> > > > causing some performance issues. This is how I am thinking of
> bringing
> > it
> > > > down:
> > > >
> > > > increase the hbase.hregion.max.filesize from 500 MB to 2 GB
> > > >
> > > > And the rebuild the HBASE table. I am assuming if after table
> rebuild,
> > I
> > > > should see the no. of regions come down to more than half. I would
> > > > basically like to stay within HBASE suggested no. of regions per RS
> > which
> > > > is about 50-200.
> > > >
> > > > Please suggest if someone has any experience doing it.
> > > >
> > > > Thanks
> > > > Upen
> > > >
> > >
> >
>
Re: Rebuild HBASE Table to reduce Regions per RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
Nice. Ted. Is there any reason why we can't do it online?
On Tue, Jan 14, 2014 at 10:47 AM, Ted Yu <yu...@gmail.com> wrote:
> Upender:
> For 15.2.2 Merge, please note the following condition:
>
> LOG.info("Verifying that HBase is not running...");
> try {
> HBaseAdmin.checkHBaseAvailable(getConf());
> LOG.fatal("HBase cluster must be off-line.");
>
> Cheers
>
>
> On Tue, Jan 14, 2014 at 10:40 AM, Vladimir Rodionov
> <vl...@gmail.com>wrote:
>
> > I have never tried this before but I think the following should work:
> >
> > 1. Alter your table:
> >
> > habse> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '5000000000'
> > (place your own number here)
> >
> > 2. Merge regions:
> >
> > http://hbase.apache.org/book/ops.regionmgt.html
> >
> >
> >
> >
> > On Tue, Jan 14, 2014 at 7:21 AM, Upender Nimbekar <
> > nimbekar.upender@gmail.com> wrote:
> >
> > > Hi,
> > > Does anyone have any experience rebuidling the HBASE table to reduce
> the
> > > number of regions. I am currently dealing with a situation where the
> no.
> > of
> > > regions per RS have gone up quite significantly (500 per RS) and
> thereby
> > > causing some performance issues. This is how I am thinking of bringing
> it
> > > down:
> > >
> > > increase the hbase.hregion.max.filesize from 500 MB to 2 GB
> > >
> > > And the rebuild the HBASE table. I am assuming if after table rebuild,
> I
> > > should see the no. of regions come down to more than half. I would
> > > basically like to stay within HBASE suggested no. of regions per RS
> which
> > > is about 50-200.
> > >
> > > Please suggest if someone has any experience doing it.
> > >
> > > Thanks
> > > Upen
> > >
> >
>
Re: Rebuild HBASE Table to reduce Regions per RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
Nice. Ted. Is there any reason why we can't do it online?
On Tue, Jan 14, 2014 at 10:47 AM, Ted Yu <yu...@gmail.com> wrote:
> Upender:
> For 15.2.2 Merge, please note the following condition:
>
> LOG.info("Verifying that HBase is not running...");
> try {
> HBaseAdmin.checkHBaseAvailable(getConf());
> LOG.fatal("HBase cluster must be off-line.");
>
> Cheers
>
>
> On Tue, Jan 14, 2014 at 10:40 AM, Vladimir Rodionov
> <vl...@gmail.com>wrote:
>
> > I have never tried this before but I think the following should work:
> >
> > 1. Alter your table:
> >
> > habse> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '5000000000'
> > (place your own number here)
> >
> > 2. Merge regions:
> >
> > http://hbase.apache.org/book/ops.regionmgt.html
> >
> >
> >
> >
> > On Tue, Jan 14, 2014 at 7:21 AM, Upender Nimbekar <
> > nimbekar.upender@gmail.com> wrote:
> >
> > > Hi,
> > > Does anyone have any experience rebuidling the HBASE table to reduce
> the
> > > number of regions. I am currently dealing with a situation where the
> no.
> > of
> > > regions per RS have gone up quite significantly (500 per RS) and
> thereby
> > > causing some performance issues. This is how I am thinking of bringing
> it
> > > down:
> > >
> > > increase the hbase.hregion.max.filesize from 500 MB to 2 GB
> > >
> > > And the rebuild the HBASE table. I am assuming if after table rebuild,
> I
> > > should see the no. of regions come down to more than half. I would
> > > basically like to stay within HBASE suggested no. of regions per RS
> which
> > > is about 50-200.
> > >
> > > Please suggest if someone has any experience doing it.
> > >
> > > Thanks
> > > Upen
> > >
> >
>
Re: Rebuild HBASE Table to reduce Regions per RS
Posted by Doug Meil <do...@explorysmedical.com>.
Normally when people are doing merges they are scripting it (I.E., not
typing out or copying/pasting each region name), but if a table is that
far gone the least problematic way is to re-create assuming you have the
capacity (i.e., not just storage, but nodes to support all the extra hbase
regions before dropping the old table).
On 1/14/14 3:38 PM, "Upender Nimbekar" <ni...@gmail.com> wrote:
>Is there a way I can Merge w/o explicitly mentioning the region names. I
>have got a total of 50000 regions. There's no way I am going to name each
>and every region. Otherway is to MIRROR a new table. Use Map Reduce job to
>copy data from old table. Delete Old Table. And Rename new table. What do
>you think. I was almost inclined to used MERGE but looks like that's not
>the way.
>
>Thanks
>Upender
>
>
>On Tue, Jan 14, 2014 at 1:47 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Upender:
>> For 15.2.2 Merge, please note the following condition:
>>
>> LOG.info("Verifying that HBase is not running...");
>> try {
>> HBaseAdmin.checkHBaseAvailable(getConf());
>> LOG.fatal("HBase cluster must be off-line.");
>>
>> Cheers
>>
>>
>> On Tue, Jan 14, 2014 at 10:40 AM, Vladimir Rodionov
>> <vl...@gmail.com>wrote:
>>
>> > I have never tried this before but I think the following should work:
>> >
>> > 1. Alter your table:
>> >
>> > habse> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '5000000000'
>> > (place your own number here)
>> >
>> > 2. Merge regions:
>> >
>> > http://hbase.apache.org/book/ops.regionmgt.html
>> >
>> >
>> >
>> >
>> > On Tue, Jan 14, 2014 at 7:21 AM, Upender Nimbekar <
>> > nimbekar.upender@gmail.com> wrote:
>> >
>> > > Hi,
>> > > Does anyone have any experience rebuidling the HBASE table to reduce
>> the
>> > > number of regions. I am currently dealing with a situation where the
>> no.
>> > of
>> > > regions per RS have gone up quite significantly (500 per RS) and
>> thereby
>> > > causing some performance issues. This is how I am thinking of
>>bringing
>> it
>> > > down:
>> > >
>> > > increase the hbase.hregion.max.filesize from 500 MB to 2 GB
>> > >
>> > > And the rebuild the HBASE table. I am assuming if after table
>>rebuild,
>> I
>> > > should see the no. of regions come down to more than half. I would
>> > > basically like to stay within HBASE suggested no. of regions per RS
>> which
>> > > is about 50-200.
>> > >
>> > > Please suggest if someone has any experience doing it.
>> > >
>> > > Thanks
>> > > Upen
>> > >
>> >
>>
Re: Rebuild HBASE Table to reduce Regions per RS
Posted by Upender Nimbekar <ni...@gmail.com>.
Is there a way I can Merge w/o explicitly mentioning the region names. I
have got a total of 50000 regions. There's no way I am going to name each
and every region. Otherway is to MIRROR a new table. Use Map Reduce job to
copy data from old table. Delete Old Table. And Rename new table. What do
you think. I was almost inclined to used MERGE but looks like that's not
the way.
Thanks
Upender
On Tue, Jan 14, 2014 at 1:47 PM, Ted Yu <yu...@gmail.com> wrote:
> Upender:
> For 15.2.2 Merge, please note the following condition:
>
> LOG.info("Verifying that HBase is not running...");
> try {
> HBaseAdmin.checkHBaseAvailable(getConf());
> LOG.fatal("HBase cluster must be off-line.");
>
> Cheers
>
>
> On Tue, Jan 14, 2014 at 10:40 AM, Vladimir Rodionov
> <vl...@gmail.com>wrote:
>
> > I have never tried this before but I think the following should work:
> >
> > 1. Alter your table:
> >
> > habse> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '5000000000'
> > (place your own number here)
> >
> > 2. Merge regions:
> >
> > http://hbase.apache.org/book/ops.regionmgt.html
> >
> >
> >
> >
> > On Tue, Jan 14, 2014 at 7:21 AM, Upender Nimbekar <
> > nimbekar.upender@gmail.com> wrote:
> >
> > > Hi,
> > > Does anyone have any experience rebuidling the HBASE table to reduce
> the
> > > number of regions. I am currently dealing with a situation where the
> no.
> > of
> > > regions per RS have gone up quite significantly (500 per RS) and
> thereby
> > > causing some performance issues. This is how I am thinking of bringing
> it
> > > down:
> > >
> > > increase the hbase.hregion.max.filesize from 500 MB to 2 GB
> > >
> > > And the rebuild the HBASE table. I am assuming if after table rebuild,
> I
> > > should see the no. of regions come down to more than half. I would
> > > basically like to stay within HBASE suggested no. of regions per RS
> which
> > > is about 50-200.
> > >
> > > Please suggest if someone has any experience doing it.
> > >
> > > Thanks
> > > Upen
> > >
> >
>
Re: Rebuild HBASE Table to reduce Regions per RS
Posted by Ted Yu <yu...@gmail.com>.
Upender:
For 15.2.2 Merge, please note the following condition:
LOG.info("Verifying that HBase is not running...");
try {
HBaseAdmin.checkHBaseAvailable(getConf());
LOG.fatal("HBase cluster must be off-line.");
Cheers
On Tue, Jan 14, 2014 at 10:40 AM, Vladimir Rodionov
<vl...@gmail.com>wrote:
> I have never tried this before but I think the following should work:
>
> 1. Alter your table:
>
> habse> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '5000000000'
> (place your own number here)
>
> 2. Merge regions:
>
> http://hbase.apache.org/book/ops.regionmgt.html
>
>
>
>
> On Tue, Jan 14, 2014 at 7:21 AM, Upender Nimbekar <
> nimbekar.upender@gmail.com> wrote:
>
> > Hi,
> > Does anyone have any experience rebuidling the HBASE table to reduce the
> > number of regions. I am currently dealing with a situation where the no.
> of
> > regions per RS have gone up quite significantly (500 per RS) and thereby
> > causing some performance issues. This is how I am thinking of bringing it
> > down:
> >
> > increase the hbase.hregion.max.filesize from 500 MB to 2 GB
> >
> > And the rebuild the HBASE table. I am assuming if after table rebuild, I
> > should see the no. of regions come down to more than half. I would
> > basically like to stay within HBASE suggested no. of regions per RS which
> > is about 50-200.
> >
> > Please suggest if someone has any experience doing it.
> >
> > Thanks
> > Upen
> >
>
Re: Rebuild HBASE Table to reduce Regions per RS
Posted by Ted Yu <yu...@gmail.com>.
Upender:
For 15.2.2 Merge, please note the following condition:
LOG.info("Verifying that HBase is not running...");
try {
HBaseAdmin.checkHBaseAvailable(getConf());
LOG.fatal("HBase cluster must be off-line.");
Cheers
On Tue, Jan 14, 2014 at 10:40 AM, Vladimir Rodionov
<vl...@gmail.com>wrote:
> I have never tried this before but I think the following should work:
>
> 1. Alter your table:
>
> habse> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '5000000000'
> (place your own number here)
>
> 2. Merge regions:
>
> http://hbase.apache.org/book/ops.regionmgt.html
>
>
>
>
> On Tue, Jan 14, 2014 at 7:21 AM, Upender Nimbekar <
> nimbekar.upender@gmail.com> wrote:
>
> > Hi,
> > Does anyone have any experience rebuidling the HBASE table to reduce the
> > number of regions. I am currently dealing with a situation where the no.
> of
> > regions per RS have gone up quite significantly (500 per RS) and thereby
> > causing some performance issues. This is how I am thinking of bringing it
> > down:
> >
> > increase the hbase.hregion.max.filesize from 500 MB to 2 GB
> >
> > And the rebuild the HBASE table. I am assuming if after table rebuild, I
> > should see the no. of regions come down to more than half. I would
> > basically like to stay within HBASE suggested no. of regions per RS which
> > is about 50-200.
> >
> > Please suggest if someone has any experience doing it.
> >
> > Thanks
> > Upen
> >
>
Re: Rebuild HBASE Table to reduce Regions per RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
I have never tried this before but I think the following should work:
1. Alter your table:
habse> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '5000000000'
(place your own number here)
2. Merge regions:
http://hbase.apache.org/book/ops.regionmgt.html
On Tue, Jan 14, 2014 at 7:21 AM, Upender Nimbekar <
nimbekar.upender@gmail.com> wrote:
> Hi,
> Does anyone have any experience rebuidling the HBASE table to reduce the
> number of regions. I am currently dealing with a situation where the no. of
> regions per RS have gone up quite significantly (500 per RS) and thereby
> causing some performance issues. This is how I am thinking of bringing it
> down:
>
> increase the hbase.hregion.max.filesize from 500 MB to 2 GB
>
> And the rebuild the HBASE table. I am assuming if after table rebuild, I
> should see the no. of regions come down to more than half. I would
> basically like to stay within HBASE suggested no. of regions per RS which
> is about 50-200.
>
> Please suggest if someone has any experience doing it.
>
> Thanks
> Upen
>
Re: Rebuild HBASE Table to reduce Regions per RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
I have never tried this before but I think the following should work:
1. Alter your table:
habse> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '5000000000'
(place your own number here)
2. Merge regions:
http://hbase.apache.org/book/ops.regionmgt.html
On Tue, Jan 14, 2014 at 7:21 AM, Upender Nimbekar <
nimbekar.upender@gmail.com> wrote:
> Hi,
> Does anyone have any experience rebuidling the HBASE table to reduce the
> number of regions. I am currently dealing with a situation where the no. of
> regions per RS have gone up quite significantly (500 per RS) and thereby
> causing some performance issues. This is how I am thinking of bringing it
> down:
>
> increase the hbase.hregion.max.filesize from 500 MB to 2 GB
>
> And the rebuild the HBASE table. I am assuming if after table rebuild, I
> should see the no. of regions come down to more than half. I would
> basically like to stay within HBASE suggested no. of regions per RS which
> is about 50-200.
>
> Please suggest if someone has any experience doing it.
>
> Thanks
> Upen
>
Re: Rebuild HBASE Table to reduce Regions per RS
Posted by Dhaval Shah <pr...@yahoo.co.in>.
If you can afford downtime for your table, there are ways to do it. You can:
- Merge regions (requires table to be disabled atleast in some older versions and probably in newer ones too)
- Go brute force by doing an export, truncate, import (this is a little more manageable when you have a large number of regions). This however is way more resource intensive and will take longer.
If you can't afford downtime, I would suggest create another table which mirrors this one and then switch to the new one
Regards,
Dhaval
________________________________
From: Upender Nimbekar <ni...@gmail.com>
To: dev@hbase.apache.org; user@hbase.apache.org
Sent: Tuesday, 14 January 2014 10:21 AM
Subject: Rebuild HBASE Table to reduce Regions per RS
Hi,
Does anyone have any experience rebuidling the HBASE table to reduce the
number of regions. I am currently dealing with a situation where the no. of
regions per RS have gone up quite significantly (500 per RS) and thereby
causing some performance issues. This is how I am thinking of bringing it
down:
increase the hbase.hregion.max.filesize from 500 MB to 2 GB
And the rebuild the HBASE table. I am assuming if after table rebuild, I
should see the no. of regions come down to more than half. I would
basically like to stay within HBASE suggested no. of regions per RS which
is about 50-200.
Please suggest if someone has any experience doing it.
Thanks
Upen