You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Norbert Burger <no...@gmail.com> on 2012/12/04 17:42:47 UTC
regions not balanced, CDH4.1.2
We upgraded to CDH4.1.2 (contains HBASE-3373) in one of our
environments. After filling that environment with data, I was
surprised to see that regions were not balanced across regionservers
at the table level. We have restarted all regionservers at least once
here.
In [1], I see Stack's reference to temporarily adding
"hbase.master.startup.retainassign=false" and restarting nodes. Is
this a necessary step on the path to region balancing nirvana?
Norbert
[1] http://search-hadoop.com/m/MQSPEyUQIv1
Re: regions not balanced, CDH4.1.2
Posted by Norbert Burger <no...@gmail.com>.
Forcing hbase.master.loadbalance.bytable=true seems to have resolved
my balance issue. Thanks again.
Norbert
On Tue, Dec 4, 2012 at 2:56 PM, Jimmy Xiang <jx...@cloudera.com> wrote:
> Right, that's config. You can turn it on and restart the cluster.
>
> Upstream it is on by default. However, it is turned off by default in
> CDH4.1.2 to be backward compatible.
>
> Thanks,
> Jimmy
>
> On Tue, Dec 4, 2012 at 11:28 AM, Norbert Burger
> <no...@gmail.com> wrote:
>> Thanks, Jimmy. Do you mean the config
>> "hbase.master.loadbalance.bytable"? According to [1] and [2], it is
>> "true" by default.
>>
>> [1] https://issues.apache.org/jira/secure/attachment/12509174/3373.txt
>> [2] http://search-hadoop.com/m/M6z7G1PKejw
>>
>> Norbert
>>
>> On Tue, Dec 4, 2012 at 2:23 PM, Jimmy Xiang <jx...@cloudera.com> wrote:
>>> In CDH4.1.2, per table region balancing is turned off by default.
>>>
>>> You can change the configuration to turn it on.
>>>
>>> Thanks,
>>> Jimmy
>>>
>>> On Tue, Dec 4, 2012 at 11:10 AM, Ted Yu <yu...@gmail.com> wrote:
>>>> Can you give us a little more detail on how much deviation the region
>>>> counts on region servers have ?
>>>>
>>>> There is a parameter, hbase.regions.slop, with default value of 0.2
>>>> This parameter allows region count to deviate by certain percentage from
>>>> average region count.
>>>>
>>>> You can tighten the value for this parameter and see if you get better
>>>> results.
>>>>
>>>> I will also put the above summary on HBASE-3373.
>>>>
>>>> Thanks
>>>>
>>>> On Tue, Dec 4, 2012 at 8:42 AM, Norbert Burger <no...@gmail.com>wrote:
>>>>
>>>>> We upgraded to CDH4.1.2 (contains HBASE-3373) in one of our
>>>>> environments. After filling that environment with data, I was
>>>>> surprised to see that regions were not balanced across regionservers
>>>>> at the table level. We have restarted all regionservers at least once
>>>>> here.
>>>>>
>>>>> In [1], I see Stack's reference to temporarily adding
>>>>> "hbase.master.startup.retainassign=false" and restarting nodes. Is
>>>>> this a necessary step on the path to region balancing nirvana?
>>>>>
>>>>> Norbert
>>>>>
>>>>> [1] http://search-hadoop.com/m/MQSPEyUQIv1
>>>>>
Re: regions not balanced, CDH4.1.2
Posted by Jimmy Xiang <jx...@cloudera.com>.
Right, that's config. You can turn it on and restart the cluster.
Upstream it is on by default. However, it is turned off by default in
CDH4.1.2 to be backward compatible.
Thanks,
Jimmy
On Tue, Dec 4, 2012 at 11:28 AM, Norbert Burger
<no...@gmail.com> wrote:
> Thanks, Jimmy. Do you mean the config
> "hbase.master.loadbalance.bytable"? According to [1] and [2], it is
> "true" by default.
>
> [1] https://issues.apache.org/jira/secure/attachment/12509174/3373.txt
> [2] http://search-hadoop.com/m/M6z7G1PKejw
>
> Norbert
>
> On Tue, Dec 4, 2012 at 2:23 PM, Jimmy Xiang <jx...@cloudera.com> wrote:
>> In CDH4.1.2, per table region balancing is turned off by default.
>>
>> You can change the configuration to turn it on.
>>
>> Thanks,
>> Jimmy
>>
>> On Tue, Dec 4, 2012 at 11:10 AM, Ted Yu <yu...@gmail.com> wrote:
>>> Can you give us a little more detail on how much deviation the region
>>> counts on region servers have ?
>>>
>>> There is a parameter, hbase.regions.slop, with default value of 0.2
>>> This parameter allows region count to deviate by certain percentage from
>>> average region count.
>>>
>>> You can tighten the value for this parameter and see if you get better
>>> results.
>>>
>>> I will also put the above summary on HBASE-3373.
>>>
>>> Thanks
>>>
>>> On Tue, Dec 4, 2012 at 8:42 AM, Norbert Burger <no...@gmail.com>wrote:
>>>
>>>> We upgraded to CDH4.1.2 (contains HBASE-3373) in one of our
>>>> environments. After filling that environment with data, I was
>>>> surprised to see that regions were not balanced across regionservers
>>>> at the table level. We have restarted all regionservers at least once
>>>> here.
>>>>
>>>> In [1], I see Stack's reference to temporarily adding
>>>> "hbase.master.startup.retainassign=false" and restarting nodes. Is
>>>> this a necessary step on the path to region balancing nirvana?
>>>>
>>>> Norbert
>>>>
>>>> [1] http://search-hadoop.com/m/MQSPEyUQIv1
>>>>
Re: regions not balanced, CDH4.1.2
Posted by Norbert Burger <no...@gmail.com>.
Thanks, Jimmy. Do you mean the config
"hbase.master.loadbalance.bytable"? According to [1] and [2], it is
"true" by default.
[1] https://issues.apache.org/jira/secure/attachment/12509174/3373.txt
[2] http://search-hadoop.com/m/M6z7G1PKejw
Norbert
On Tue, Dec 4, 2012 at 2:23 PM, Jimmy Xiang <jx...@cloudera.com> wrote:
> In CDH4.1.2, per table region balancing is turned off by default.
>
> You can change the configuration to turn it on.
>
> Thanks,
> Jimmy
>
> On Tue, Dec 4, 2012 at 11:10 AM, Ted Yu <yu...@gmail.com> wrote:
>> Can you give us a little more detail on how much deviation the region
>> counts on region servers have ?
>>
>> There is a parameter, hbase.regions.slop, with default value of 0.2
>> This parameter allows region count to deviate by certain percentage from
>> average region count.
>>
>> You can tighten the value for this parameter and see if you get better
>> results.
>>
>> I will also put the above summary on HBASE-3373.
>>
>> Thanks
>>
>> On Tue, Dec 4, 2012 at 8:42 AM, Norbert Burger <no...@gmail.com>wrote:
>>
>>> We upgraded to CDH4.1.2 (contains HBASE-3373) in one of our
>>> environments. After filling that environment with data, I was
>>> surprised to see that regions were not balanced across regionservers
>>> at the table level. We have restarted all regionservers at least once
>>> here.
>>>
>>> In [1], I see Stack's reference to temporarily adding
>>> "hbase.master.startup.retainassign=false" and restarting nodes. Is
>>> this a necessary step on the path to region balancing nirvana?
>>>
>>> Norbert
>>>
>>> [1] http://search-hadoop.com/m/MQSPEyUQIv1
>>>
Re: regions not balanced, CDH4.1.2
Posted by Jimmy Xiang <jx...@cloudera.com>.
In CDH4.1.2, per table region balancing is turned off by default.
You can change the configuration to turn it on.
Thanks,
Jimmy
On Tue, Dec 4, 2012 at 11:10 AM, Ted Yu <yu...@gmail.com> wrote:
> Can you give us a little more detail on how much deviation the region
> counts on region servers have ?
>
> There is a parameter, hbase.regions.slop, with default value of 0.2
> This parameter allows region count to deviate by certain percentage from
> average region count.
>
> You can tighten the value for this parameter and see if you get better
> results.
>
> I will also put the above summary on HBASE-3373.
>
> Thanks
>
> On Tue, Dec 4, 2012 at 8:42 AM, Norbert Burger <no...@gmail.com>wrote:
>
>> We upgraded to CDH4.1.2 (contains HBASE-3373) in one of our
>> environments. After filling that environment with data, I was
>> surprised to see that regions were not balanced across regionservers
>> at the table level. We have restarted all regionservers at least once
>> here.
>>
>> In [1], I see Stack's reference to temporarily adding
>> "hbase.master.startup.retainassign=false" and restarting nodes. Is
>> this a necessary step on the path to region balancing nirvana?
>>
>> Norbert
>>
>> [1] http://search-hadoop.com/m/MQSPEyUQIv1
>>
Re: regions not balanced, CDH4.1.2
Posted by Ted Yu <yu...@gmail.com>.
Thanks for the information.
Please turn on per-table load balancing and tell us what it looks like.
On Tue, Dec 4, 2012 at 11:32 AM, Norbert Burger <no...@gmail.com>wrote:
> On Tue, Dec 4, 2012 at 2:10 PM, Ted Yu <yu...@gmail.com> wrote:
> > Can you give us a little more detail on how much deviation the region
> > counts on region servers have ?
> >
> > There is a parameter, hbase.regions.slop, with default value of 0.2
> > This parameter allows region count to deviate by certain percentage from
> > average region count.
> >
> > You can tighten the value for this parameter and see if you get better
> > results.
>
> Thanks for the response. I think this isn't an issue with region slop
> -- I had a table with only ~15 regions, and half of them ended up on
> one region server. I've since manually rebalanced that table, but
> here's an example from different table [1].
>
> Norbert
>
> [1]
> Regions by Region Server
> Region Server Region Count
> http://aspen8hdp9.turner.com:60030/ 25
> http://aspen8hdp10.turner.com:60030/ 25
> http://aspen8hdp16.turner.com:60030/ 38
> http://aspen8hdp13.turner.com:60030/ 26
> http://aspen8hdp15.turner.com:60030/ 26
> http://aspen8hdp12.turner.com:60030/ 43
> http://aspen8hdp14.turner.com:60030/ 31
> http://aspen8hdp11.turner.com:60030/ 42
>
Re: regions not balanced, CDH4.1.2
Posted by Norbert Burger <no...@gmail.com>.
On Tue, Dec 4, 2012 at 2:10 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you give us a little more detail on how much deviation the region
> counts on region servers have ?
>
> There is a parameter, hbase.regions.slop, with default value of 0.2
> This parameter allows region count to deviate by certain percentage from
> average region count.
>
> You can tighten the value for this parameter and see if you get better
> results.
Thanks for the response. I think this isn't an issue with region slop
-- I had a table with only ~15 regions, and half of them ended up on
one region server. I've since manually rebalanced that table, but
here's an example from different table [1].
Norbert
[1]
Regions by Region Server
Region Server Region Count
http://aspen8hdp9.turner.com:60030/ 25
http://aspen8hdp10.turner.com:60030/ 25
http://aspen8hdp16.turner.com:60030/ 38
http://aspen8hdp13.turner.com:60030/ 26
http://aspen8hdp15.turner.com:60030/ 26
http://aspen8hdp12.turner.com:60030/ 43
http://aspen8hdp14.turner.com:60030/ 31
http://aspen8hdp11.turner.com:60030/ 42
Re: regions not balanced, CDH4.1.2
Posted by Ted Yu <yu...@gmail.com>.
Can you give us a little more detail on how much deviation the region
counts on region servers have ?
There is a parameter, hbase.regions.slop, with default value of 0.2
This parameter allows region count to deviate by certain percentage from
average region count.
You can tighten the value for this parameter and see if you get better
results.
I will also put the above summary on HBASE-3373.
Thanks
On Tue, Dec 4, 2012 at 8:42 AM, Norbert Burger <no...@gmail.com>wrote:
> We upgraded to CDH4.1.2 (contains HBASE-3373) in one of our
> environments. After filling that environment with data, I was
> surprised to see that regions were not balanced across regionservers
> at the table level. We have restarted all regionservers at least once
> here.
>
> In [1], I see Stack's reference to temporarily adding
> "hbase.master.startup.retainassign=false" and restarting nodes. Is
> this a necessary step on the path to region balancing nirvana?
>
> Norbert
>
> [1] http://search-hadoop.com/m/MQSPEyUQIv1
>