You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Nicolae Marasoiu <ni...@adswizz.com> on 2015/10/08 16:49:02 UTC
alter column family - possible operational impacts on big tables
Hi,
If we run at night an alter column family, set ttl, my understanding is that it will disable the table, make the alter, and re-enable the table, which can be some time for large tables with 30+ regions (hbase version 0.94 [☹] ).
Do you have any advice about this? How long can it take per region? What is the operational hit at the time of the alter command being issued, and what when compaction runs on the table? I imagine that compaction is not too affected by this, just by filtering out more records when re-writing the new HFiles, is this correct?
Thanks,
Nicu
Re: alter column family - possible operational impacts on big tables
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
5000 regions in 10RS it's 500 regions per RS just for this table... If are
above the recommended limit. How big are your regions? I mean, MAXFILESIZE?
JM
2015-10-09 9:46 GMT-04:00 Anil Gupta <an...@gmail.com>:
> Hi Nicolas,
>
> For a table with 5k regions, it should not take more than 10 min for alter
> table operations.
> Also, in HBase 1.0+, alter table operations does not require disabling the
> table. So, you are encouraged to upgrade.
>
> Sent from my iPhone
>
> > On Oct 9, 2015, at 1:15 AM, Nicolae Marasoiu <
> nicolae.marasoiu@adswizz.com> wrote:
> >
> > Hi,
> >
> > Indeed, we have tables with 1-5000 regions, distributed on 10-15 RSs.
> >
> > A few hours are sufficient to do the alter one a single such table,
> right?
> >
> > Thanks,
> > Nicu
> >
> > ________________________________________
> > From: Jean-Marc Spaggiari <je...@spaggiari.org>
> > Sent: Thursday, October 8, 2015 10:19 PM
> > To: user
> > Subject: Re: alter column family - possible operational impacts on big
> tables
> >
> > Hi Nicu,
> >
> > Indeed, with 0.94 you have to disable the table before doing the alter.
> > However, for 30 regions, it should be pretty fast. When you say 30+, are
> > you talking about like 1K regions? Or more like 32? The alter will only
> > update the meta table, so not that much impact on the servers. And no
> > compactions required for that. The ttl will only take effect at the next
> > compaction by, as you said, filtering out more records.
> >
> > JM
> >
> > 2015-10-08 10:49 GMT-04:00 Nicolae Marasoiu <
> nicolae.marasoiu@adswizz.com>:
> >
> >> Hi,
> >>
> >>
> >> If we run at night an alter column family, set ttl, my understanding is
> >> that it will disable the table, make the alter, and re-enable the table,
> >> which can be some time for large tables with 30+ regions (hbase version
> >> 0.94 [image: ☹] ).
> >>
> >>
> >> Do you have any advice about this? How long can it take per region? What
> >> is the operational hit at the time of the alter command being issued,
> and
> >> what when compaction runs on the table? I imagine that compaction is not
> >> too affected by this, just by filtering out more records when re-writing
> >> the new HFiles, is this correct?
> >>
> >>
> >> Thanks,
> >>
> >> Nicu
> >>
>
Re: alter column family - possible operational impacts on big tables
Posted by Ted Yu <yu...@gmail.com>.
I agree with Anil w.r.t. upgrade.
Nicolae:
If you can afford a few hours for the table to be offline, it seems
upgrading to 1.x first would be more beneficial.
On Fri, Oct 9, 2015 at 6:46 AM, Anil Gupta <an...@gmail.com> wrote:
> Hi Nicolas,
>
> For a table with 5k regions, it should not take more than 10 min for alter
> table operations.
> Also, in HBase 1.0+, alter table operations does not require disabling the
> table. So, you are encouraged to upgrade.
>
> Sent from my iPhone
>
> > On Oct 9, 2015, at 1:15 AM, Nicolae Marasoiu <
> nicolae.marasoiu@adswizz.com> wrote:
> >
> > Hi,
> >
> > Indeed, we have tables with 1-5000 regions, distributed on 10-15 RSs.
> >
> > A few hours are sufficient to do the alter one a single such table,
> right?
> >
> > Thanks,
> > Nicu
> >
> > ________________________________________
> > From: Jean-Marc Spaggiari <je...@spaggiari.org>
> > Sent: Thursday, October 8, 2015 10:19 PM
> > To: user
> > Subject: Re: alter column family - possible operational impacts on big
> tables
> >
> > Hi Nicu,
> >
> > Indeed, with 0.94 you have to disable the table before doing the alter.
> > However, for 30 regions, it should be pretty fast. When you say 30+, are
> > you talking about like 1K regions? Or more like 32? The alter will only
> > update the meta table, so not that much impact on the servers. And no
> > compactions required for that. The ttl will only take effect at the next
> > compaction by, as you said, filtering out more records.
> >
> > JM
> >
> > 2015-10-08 10:49 GMT-04:00 Nicolae Marasoiu <
> nicolae.marasoiu@adswizz.com>:
> >
> >> Hi,
> >>
> >>
> >> If we run at night an alter column family, set ttl, my understanding is
> >> that it will disable the table, make the alter, and re-enable the table,
> >> which can be some time for large tables with 30+ regions (hbase version
> >> 0.94 [image: ☹] ).
> >>
> >>
> >> Do you have any advice about this? How long can it take per region? What
> >> is the operational hit at the time of the alter command being issued,
> and
> >> what when compaction runs on the table? I imagine that compaction is not
> >> too affected by this, just by filtering out more records when re-writing
> >> the new HFiles, is this correct?
> >>
> >>
> >> Thanks,
> >>
> >> Nicu
> >>
>
Re: alter column family - possible operational impacts on big tables
Posted by Anil Gupta <an...@gmail.com>.
Hi Nicolas,
For a table with 5k regions, it should not take more than 10 min for alter table operations.
Also, in HBase 1.0+, alter table operations does not require disabling the table. So, you are encouraged to upgrade.
Sent from my iPhone
> On Oct 9, 2015, at 1:15 AM, Nicolae Marasoiu <ni...@adswizz.com> wrote:
>
> Hi,
>
> Indeed, we have tables with 1-5000 regions, distributed on 10-15 RSs.
>
> A few hours are sufficient to do the alter one a single such table, right?
>
> Thanks,
> Nicu
>
> ________________________________________
> From: Jean-Marc Spaggiari <je...@spaggiari.org>
> Sent: Thursday, October 8, 2015 10:19 PM
> To: user
> Subject: Re: alter column family - possible operational impacts on big tables
>
> Hi Nicu,
>
> Indeed, with 0.94 you have to disable the table before doing the alter.
> However, for 30 regions, it should be pretty fast. When you say 30+, are
> you talking about like 1K regions? Or more like 32? The alter will only
> update the meta table, so not that much impact on the servers. And no
> compactions required for that. The ttl will only take effect at the next
> compaction by, as you said, filtering out more records.
>
> JM
>
> 2015-10-08 10:49 GMT-04:00 Nicolae Marasoiu <ni...@adswizz.com>:
>
>> Hi,
>>
>>
>> If we run at night an alter column family, set ttl, my understanding is
>> that it will disable the table, make the alter, and re-enable the table,
>> which can be some time for large tables with 30+ regions (hbase version
>> 0.94 [image: ☹] ).
>>
>>
>> Do you have any advice about this? How long can it take per region? What
>> is the operational hit at the time of the alter command being issued, and
>> what when compaction runs on the table? I imagine that compaction is not
>> too affected by this, just by filtering out more records when re-writing
>> the new HFiles, is this correct?
>>
>>
>> Thanks,
>>
>> Nicu
>>
Re: alter column family - possible operational impacts on big tables
Posted by Nicolae Marasoiu <ni...@adswizz.com>.
Hi,
Indeed, we have tables with 1-5000 regions, distributed on 10-15 RSs.
A few hours are sufficient to do the alter one a single such table, right?
Thanks,
Nicu
________________________________________
From: Jean-Marc Spaggiari <je...@spaggiari.org>
Sent: Thursday, October 8, 2015 10:19 PM
To: user
Subject: Re: alter column family - possible operational impacts on big tables
Hi Nicu,
Indeed, with 0.94 you have to disable the table before doing the alter.
However, for 30 regions, it should be pretty fast. When you say 30+, are
you talking about like 1K regions? Or more like 32? The alter will only
update the meta table, so not that much impact on the servers. And no
compactions required for that. The ttl will only take effect at the next
compaction by, as you said, filtering out more records.
JM
2015-10-08 10:49 GMT-04:00 Nicolae Marasoiu <ni...@adswizz.com>:
> Hi,
>
>
> If we run at night an alter column family, set ttl, my understanding is
> that it will disable the table, make the alter, and re-enable the table,
> which can be some time for large tables with 30+ regions (hbase version
> 0.94 [image: ☹] ).
>
>
> Do you have any advice about this? How long can it take per region? What
> is the operational hit at the time of the alter command being issued, and
> what when compaction runs on the table? I imagine that compaction is not
> too affected by this, just by filtering out more records when re-writing
> the new HFiles, is this correct?
>
>
> Thanks,
>
> Nicu
>
Re: alter column family - possible operational impacts on big tables
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Nicu,
Indeed, with 0.94 you have to disable the table before doing the alter.
However, for 30 regions, it should be pretty fast. When you say 30+, are
you talking about like 1K regions? Or more like 32? The alter will only
update the meta table, so not that much impact on the servers. And no
compactions required for that. The ttl will only take effect at the next
compaction by, as you said, filtering out more records.
JM
2015-10-08 10:49 GMT-04:00 Nicolae Marasoiu <ni...@adswizz.com>:
> Hi,
>
>
> If we run at night an alter column family, set ttl, my understanding is
> that it will disable the table, make the alter, and re-enable the table,
> which can be some time for large tables with 30+ regions (hbase version
> 0.94 [image: ☹] ).
>
>
> Do you have any advice about this? How long can it take per region? What
> is the operational hit at the time of the alter command being issued, and
> what when compaction runs on the table? I imagine that compaction is not
> too affected by this, just by filtering out more records when re-writing
> the new HFiles, is this correct?
>
>
> Thanks,
>
> Nicu
>