You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by WangRamon <ra...@hotmail.com> on 2013/11/05 10:56:14 UTC
How to change split policy of a specific existing table(not global)?
Hi Folks
I'm trying to change the default split policy of an existing table in HBase 0.94.6-cdh4.4.0. The way i think should work is by using HBase Shell, so i executed the following command in the SHELL:
disable 'SPLIT_TEST_BIG'alter 'SPLIT_TEST_BIG', {METHOD => 'table_att', 'SPLIT_POLICY' => 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}alter 'SPLIT_TEST_BIG', {METHOD => 'table_att', 'MAX_FILESIZE' => '107374182400'}enable 'SPLIT_TEST_BIG'
But it doesn't work, only the 'MAX_FILESIZE' property get changed, the table still uses the default split policy which is 'IncreasingToUpperBoundRegionSplitPolicy', so can anyone tell me what is the correct way to do it, thanks!
CheersRamon
RE: How to change split policy of a specific existing table(not global)?
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
Shel does not support all table/column families attributes in 0.94.6
This should work:
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor desc = admin.getTableDescriptor(Bytes.toBytes(SPLIT_TEST_BIG));// get your table descriptor
admin.disableTable(SPLIT_TEST_BIG);
desc.setValue(HTableDescriptor.SPLIT_POLICY, "org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy");
admin.modifyTable(Bytes.toBytes(SPLIT_TEST_BIG), desc);
admin.enableTable(SPLIT_TEST_BIG);
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: WangRamon [ramon_wang@hotmail.com]
Sent: Tuesday, November 05, 2013 1:56 AM
To: user@hbase.apache.org
Subject: How to change split policy of a specific existing table(not global)?
Hi Folks
I'm trying to change the default split policy of an existing table in HBase 0.94.6-cdh4.4.0. The way i think should work is by using HBase Shell, so i executed the following command in the SHELL:
disable 'SPLIT_TEST_BIG'alter 'SPLIT_TEST_BIG', {METHOD => 'table_att', 'SPLIT_POLICY' => 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}alter 'SPLIT_TEST_BIG', {METHOD => 'table_att', 'MAX_FILESIZE' => '107374182400'}enable 'SPLIT_TEST_BIG'
But it doesn't work, only the 'MAX_FILESIZE' property get changed, the table still uses the default split policy which is 'IncreasingToUpperBoundRegionSplitPolicy', so can anyone tell me what is the correct way to do it, thanks!
CheersRamon
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
RE: How to change split policy of a specific existing table(not global)?
Posted by WangRamon <ra...@hotmail.com>.
Thanks Vladimir, you're right, the ruby based Shell cannot support custom parameters(at least for 0.94.6-cdh4.4.0). I added some debug log in the HBase source code which also proved it.
The java code does work. I also added comments in the ticket.
CheersRamon
> From: vrodionov@carrieriq.com
> To: user@hbase.apache.org
> Date: Tue, 5 Nov 2013 09:43:40 -0800
> Subject: RE: How to change split policy of a specific existing table(not global)?
>
> There is a ticket for that :)
>
> https://issues.apache.org/jira/browse/HBASE-9218
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: saint.ack@gmail.com [saint.ack@gmail.com] On Behalf Of Stack [stack@duboce.net]
> Sent: Tuesday, November 05, 2013 7:53 AM
> To: Hbase-User
> Subject: Re: How to change split policy of a specific existing table(not global)?
>
> On Tue, Nov 5, 2013 at 6:04 AM, WangRamon <ra...@hotmail.com> wrote:
>
> > Hi JM
> > Thanks for your help, yes, as you said the syntax is totally correct as it
> > can be executed in HBase Shell without any errors.
> > By checking the source code of class HRegion which initialise the region
> > policy from HTableDescriptor, not sure if that SPLIT_POLICY value can be
> > dynamically changed after a table(and its first child region) is created?
> >
> > > 1) If you look in the Web UI, do you see you new attribute set for this
> > > table?
> > I don't know there is a specific UI which i can view the new attributes
> > except the master-status UI, if that's the UI you are talking about, i can
> > only see the MAX_FILESIZE property is set correctly:{NAME =>
> > 'SPLIT_TEST_BIG', MAX_FILESIZE => '107374182400', FAMILIES => [{NAME =>
> > 'f1'}]}
> >
>
> What JMS said.
>
> Seems like you are looking at the right place in the UI.
>
> After altering the table, if you do
>
> hbase> describe "SPLIT_TEST_BIG"
>
> ... do you see the new attribute set?
>
> If not, the shell must be not setting this attribute properly (bug).
>
> St.Ack
>
>
>
>
>
> > > 2) How have you verified that it doesn't work? What do you have on the
> > logs?
> > I use YCSB to insert data into that table, and i did saw there are some
> > new regions created after i running the test for a while. And here is the
> > LOG i get from one of the region severs:
> > 2013-11-05 08:34:46,571 DEBUG
> > org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy:
> > ShouldSplit because f1 size=414926238, sizeToCheck=134217728,
> > regionsWithCommonTable=1
> > Any idea?
> > ThanksRamon
> >
> > > From: jean-marc@spaggiari.org
> > > Date: Tue, 5 Nov 2013 07:40:04 -0500
> > > Subject: Re: How to change split policy of a specific existing table(not
> > global)?
> > > To: user@hbase.apache.org
> > >
> > > This seems to be correct.
> > >
> > > 2 questions.
> > >
> > > 1) If you look in the Web UI, do you see you new attribute set for this
> > > table?
> > > 2) How have you verified that it doesn't work? What do you have on the
> > logs?
> > >
> > > JM
> > >
> > >
> > > 2013/11/5 WangRamon <ra...@hotmail.com>
> > >
> > > > Hi Folks
> > > > I'm trying to change the default split policy of an existing table in
> > > > HBase 0.94.6-cdh4.4.0. The way i think should work is by using HBase
> > Shell,
> > > > so i executed the following command in the SHELL:
> > > > disable 'SPLIT_TEST_BIG'alter 'SPLIT_TEST_BIG', {METHOD => 'table_att',
> > > > 'SPLIT_POLICY' =>
> > > >
> > 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}alter
> > > > 'SPLIT_TEST_BIG', {METHOD => 'table_att', 'MAX_FILESIZE' =>
> > > > '107374182400'}enable 'SPLIT_TEST_BIG'
> > > >
> > > > But it doesn't work, only the 'MAX_FILESIZE' property get changed, the
> > > > table still uses the default split policy which is
> > > > 'IncreasingToUpperBoundRegionSplitPolicy', so can anyone tell me what
> > is
> > > > the correct way to do it, thanks!
> > > > CheersRamon
> >
> >
>
> Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
RE: How to change split policy of a specific existing table(not global)?
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
There is a ticket for that :)
https://issues.apache.org/jira/browse/HBASE-9218
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: saint.ack@gmail.com [saint.ack@gmail.com] On Behalf Of Stack [stack@duboce.net]
Sent: Tuesday, November 05, 2013 7:53 AM
To: Hbase-User
Subject: Re: How to change split policy of a specific existing table(not global)?
On Tue, Nov 5, 2013 at 6:04 AM, WangRamon <ra...@hotmail.com> wrote:
> Hi JM
> Thanks for your help, yes, as you said the syntax is totally correct as it
> can be executed in HBase Shell without any errors.
> By checking the source code of class HRegion which initialise the region
> policy from HTableDescriptor, not sure if that SPLIT_POLICY value can be
> dynamically changed after a table(and its first child region) is created?
>
> > 1) If you look in the Web UI, do you see you new attribute set for this
> > table?
> I don't know there is a specific UI which i can view the new attributes
> except the master-status UI, if that's the UI you are talking about, i can
> only see the MAX_FILESIZE property is set correctly:{NAME =>
> 'SPLIT_TEST_BIG', MAX_FILESIZE => '107374182400', FAMILIES => [{NAME =>
> 'f1'}]}
>
What JMS said.
Seems like you are looking at the right place in the UI.
After altering the table, if you do
hbase> describe "SPLIT_TEST_BIG"
... do you see the new attribute set?
If not, the shell must be not setting this attribute properly (bug).
St.Ack
> > 2) How have you verified that it doesn't work? What do you have on the
> logs?
> I use YCSB to insert data into that table, and i did saw there are some
> new regions created after i running the test for a while. And here is the
> LOG i get from one of the region severs:
> 2013-11-05 08:34:46,571 DEBUG
> org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy:
> ShouldSplit because f1 size=414926238, sizeToCheck=134217728,
> regionsWithCommonTable=1
> Any idea?
> ThanksRamon
>
> > From: jean-marc@spaggiari.org
> > Date: Tue, 5 Nov 2013 07:40:04 -0500
> > Subject: Re: How to change split policy of a specific existing table(not
> global)?
> > To: user@hbase.apache.org
> >
> > This seems to be correct.
> >
> > 2 questions.
> >
> > 1) If you look in the Web UI, do you see you new attribute set for this
> > table?
> > 2) How have you verified that it doesn't work? What do you have on the
> logs?
> >
> > JM
> >
> >
> > 2013/11/5 WangRamon <ra...@hotmail.com>
> >
> > > Hi Folks
> > > I'm trying to change the default split policy of an existing table in
> > > HBase 0.94.6-cdh4.4.0. The way i think should work is by using HBase
> Shell,
> > > so i executed the following command in the SHELL:
> > > disable 'SPLIT_TEST_BIG'alter 'SPLIT_TEST_BIG', {METHOD => 'table_att',
> > > 'SPLIT_POLICY' =>
> > >
> 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}alter
> > > 'SPLIT_TEST_BIG', {METHOD => 'table_att', 'MAX_FILESIZE' =>
> > > '107374182400'}enable 'SPLIT_TEST_BIG'
> > >
> > > But it doesn't work, only the 'MAX_FILESIZE' property get changed, the
> > > table still uses the default split policy which is
> > > 'IncreasingToUpperBoundRegionSplitPolicy', so can anyone tell me what
> is
> > > the correct way to do it, thanks!
> > > CheersRamon
>
>
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
RE: How to change split policy of a specific existing table(not global)?
Posted by WangRamon <ra...@hotmail.com>.
Hi JM
I didn't see any new attributes after running the describe command:
hbase(main):004:0> describe "SPLIT_TEST_BIG"DESCRIPTION {NAME => 'SPLIT_TEST_BIG', MAX_FILESIZE => '10737418240 0', FAMILIES => [{NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0' , VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'fal se', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE _ON_DISK => 'true', BLOCKCACHE => 'true'}]}1 row(s) in 0.0510 seconds
So for now, i'm trying to compile the CDH4.4 HBase (add some of the debug message), but got some errors during the mvn package process.... Do you happen to use CDH4.4 and know how to compile it? ^__^
ThanksRamon
> Date: Tue, 5 Nov 2013 07:53:36 -0800
> Subject: Re: How to change split policy of a specific existing table(not global)?
> From: stack@duboce.net
> To: user@hbase.apache.org
>
> On Tue, Nov 5, 2013 at 6:04 AM, WangRamon <ra...@hotmail.com> wrote:
>
> > Hi JM
> > Thanks for your help, yes, as you said the syntax is totally correct as it
> > can be executed in HBase Shell without any errors.
> > By checking the source code of class HRegion which initialise the region
> > policy from HTableDescriptor, not sure if that SPLIT_POLICY value can be
> > dynamically changed after a table(and its first child region) is created?
> >
> > > 1) If you look in the Web UI, do you see you new attribute set for this
> > > table?
> > I don't know there is a specific UI which i can view the new attributes
> > except the master-status UI, if that's the UI you are talking about, i can
> > only see the MAX_FILESIZE property is set correctly:{NAME =>
> > 'SPLIT_TEST_BIG', MAX_FILESIZE => '107374182400', FAMILIES => [{NAME =>
> > 'f1'}]}
> >
>
> What JMS said.
>
> Seems like you are looking at the right place in the UI.
>
> After altering the table, if you do
>
> hbase> describe "SPLIT_TEST_BIG"
>
> ... do you see the new attribute set?
>
> If not, the shell must be not setting this attribute properly (bug).
>
> St.Ack
>
>
>
>
>
> > > 2) How have you verified that it doesn't work? What do you have on the
> > logs?
> > I use YCSB to insert data into that table, and i did saw there are some
> > new regions created after i running the test for a while. And here is the
> > LOG i get from one of the region severs:
> > 2013-11-05 08:34:46,571 DEBUG
> > org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy:
> > ShouldSplit because f1 size=414926238, sizeToCheck=134217728,
> > regionsWithCommonTable=1
> > Any idea?
> > ThanksRamon
> >
> > > From: jean-marc@spaggiari.org
> > > Date: Tue, 5 Nov 2013 07:40:04 -0500
> > > Subject: Re: How to change split policy of a specific existing table(not
> > global)?
> > > To: user@hbase.apache.org
> > >
> > > This seems to be correct.
> > >
> > > 2 questions.
> > >
> > > 1) If you look in the Web UI, do you see you new attribute set for this
> > > table?
> > > 2) How have you verified that it doesn't work? What do you have on the
> > logs?
> > >
> > > JM
> > >
> > >
> > > 2013/11/5 WangRamon <ra...@hotmail.com>
> > >
> > > > Hi Folks
> > > > I'm trying to change the default split policy of an existing table in
> > > > HBase 0.94.6-cdh4.4.0. The way i think should work is by using HBase
> > Shell,
> > > > so i executed the following command in the SHELL:
> > > > disable 'SPLIT_TEST_BIG'alter 'SPLIT_TEST_BIG', {METHOD => 'table_att',
> > > > 'SPLIT_POLICY' =>
> > > >
> > 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}alter
> > > > 'SPLIT_TEST_BIG', {METHOD => 'table_att', 'MAX_FILESIZE' =>
> > > > '107374182400'}enable 'SPLIT_TEST_BIG'
> > > >
> > > > But it doesn't work, only the 'MAX_FILESIZE' property get changed, the
> > > > table still uses the default split policy which is
> > > > 'IncreasingToUpperBoundRegionSplitPolicy', so can anyone tell me what
> > is
> > > > the correct way to do it, thanks!
> > > > CheersRamon
> >
> >
Re: How to change split policy of a specific existing table(not global)?
Posted by Stack <st...@duboce.net>.
On Tue, Nov 5, 2013 at 6:04 AM, WangRamon <ra...@hotmail.com> wrote:
> Hi JM
> Thanks for your help, yes, as you said the syntax is totally correct as it
> can be executed in HBase Shell without any errors.
> By checking the source code of class HRegion which initialise the region
> policy from HTableDescriptor, not sure if that SPLIT_POLICY value can be
> dynamically changed after a table(and its first child region) is created?
>
> > 1) If you look in the Web UI, do you see you new attribute set for this
> > table?
> I don't know there is a specific UI which i can view the new attributes
> except the master-status UI, if that's the UI you are talking about, i can
> only see the MAX_FILESIZE property is set correctly:{NAME =>
> 'SPLIT_TEST_BIG', MAX_FILESIZE => '107374182400', FAMILIES => [{NAME =>
> 'f1'}]}
>
What JMS said.
Seems like you are looking at the right place in the UI.
After altering the table, if you do
hbase> describe "SPLIT_TEST_BIG"
... do you see the new attribute set?
If not, the shell must be not setting this attribute properly (bug).
St.Ack
> > 2) How have you verified that it doesn't work? What do you have on the
> logs?
> I use YCSB to insert data into that table, and i did saw there are some
> new regions created after i running the test for a while. And here is the
> LOG i get from one of the region severs:
> 2013-11-05 08:34:46,571 DEBUG
> org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy:
> ShouldSplit because f1 size=414926238, sizeToCheck=134217728,
> regionsWithCommonTable=1
> Any idea?
> ThanksRamon
>
> > From: jean-marc@spaggiari.org
> > Date: Tue, 5 Nov 2013 07:40:04 -0500
> > Subject: Re: How to change split policy of a specific existing table(not
> global)?
> > To: user@hbase.apache.org
> >
> > This seems to be correct.
> >
> > 2 questions.
> >
> > 1) If you look in the Web UI, do you see you new attribute set for this
> > table?
> > 2) How have you verified that it doesn't work? What do you have on the
> logs?
> >
> > JM
> >
> >
> > 2013/11/5 WangRamon <ra...@hotmail.com>
> >
> > > Hi Folks
> > > I'm trying to change the default split policy of an existing table in
> > > HBase 0.94.6-cdh4.4.0. The way i think should work is by using HBase
> Shell,
> > > so i executed the following command in the SHELL:
> > > disable 'SPLIT_TEST_BIG'alter 'SPLIT_TEST_BIG', {METHOD => 'table_att',
> > > 'SPLIT_POLICY' =>
> > >
> 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}alter
> > > 'SPLIT_TEST_BIG', {METHOD => 'table_att', 'MAX_FILESIZE' =>
> > > '107374182400'}enable 'SPLIT_TEST_BIG'
> > >
> > > But it doesn't work, only the 'MAX_FILESIZE' property get changed, the
> > > table still uses the default split policy which is
> > > 'IncreasingToUpperBoundRegionSplitPolicy', so can anyone tell me what
> is
> > > the correct way to do it, thanks!
> > > CheersRamon
>
>
RE: How to change split policy of a specific existing table(not global)?
Posted by WangRamon <ra...@hotmail.com>.
Hi JM
Thanks for your help, yes, as you said the syntax is totally correct as it can be executed in HBase Shell without any errors.
By checking the source code of class HRegion which initialise the region policy from HTableDescriptor, not sure if that SPLIT_POLICY value can be dynamically changed after a table(and its first child region) is created?
> 1) If you look in the Web UI, do you see you new attribute set for this
> table?
I don't know there is a specific UI which i can view the new attributes except the master-status UI, if that's the UI you are talking about, i can only see the MAX_FILESIZE property is set correctly:{NAME => 'SPLIT_TEST_BIG', MAX_FILESIZE => '107374182400', FAMILIES => [{NAME => 'f1'}]}
> 2) How have you verified that it doesn't work? What do you have on the logs?
I use YCSB to insert data into that table, and i did saw there are some new regions created after i running the test for a while. And here is the LOG i get from one of the region severs:
2013-11-05 08:34:46,571 DEBUG org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because f1 size=414926238, sizeToCheck=134217728, regionsWithCommonTable=1
Any idea?
ThanksRamon
> From: jean-marc@spaggiari.org
> Date: Tue, 5 Nov 2013 07:40:04 -0500
> Subject: Re: How to change split policy of a specific existing table(not global)?
> To: user@hbase.apache.org
>
> This seems to be correct.
>
> 2 questions.
>
> 1) If you look in the Web UI, do you see you new attribute set for this
> table?
> 2) How have you verified that it doesn't work? What do you have on the logs?
>
> JM
>
>
> 2013/11/5 WangRamon <ra...@hotmail.com>
>
> > Hi Folks
> > I'm trying to change the default split policy of an existing table in
> > HBase 0.94.6-cdh4.4.0. The way i think should work is by using HBase Shell,
> > so i executed the following command in the SHELL:
> > disable 'SPLIT_TEST_BIG'alter 'SPLIT_TEST_BIG', {METHOD => 'table_att',
> > 'SPLIT_POLICY' =>
> > 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}alter
> > 'SPLIT_TEST_BIG', {METHOD => 'table_att', 'MAX_FILESIZE' =>
> > '107374182400'}enable 'SPLIT_TEST_BIG'
> >
> > But it doesn't work, only the 'MAX_FILESIZE' property get changed, the
> > table still uses the default split policy which is
> > 'IncreasingToUpperBoundRegionSplitPolicy', so can anyone tell me what is
> > the correct way to do it, thanks!
> > CheersRamon
Re: How to change split policy of a specific existing table(not global)?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
This seems to be correct.
2 questions.
1) If you look in the Web UI, do you see you new attribute set for this
table?
2) How have you verified that it doesn't work? What do you have on the logs?
JM
2013/11/5 WangRamon <ra...@hotmail.com>
> Hi Folks
> I'm trying to change the default split policy of an existing table in
> HBase 0.94.6-cdh4.4.0. The way i think should work is by using HBase Shell,
> so i executed the following command in the SHELL:
> disable 'SPLIT_TEST_BIG'alter 'SPLIT_TEST_BIG', {METHOD => 'table_att',
> 'SPLIT_POLICY' =>
> 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}alter
> 'SPLIT_TEST_BIG', {METHOD => 'table_att', 'MAX_FILESIZE' =>
> '107374182400'}enable 'SPLIT_TEST_BIG'
>
> But it doesn't work, only the 'MAX_FILESIZE' property get changed, the
> table still uses the default split policy which is
> 'IncreasingToUpperBoundRegionSplitPolicy', so can anyone tell me what is
> the correct way to do it, thanks!
> CheersRamon