You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by David Charle <db...@gmail.com> on 2012/04/24 03:41:45 UTC
Unbalanced region server
Hi
What can make a region server unbalanced when it comes to space ? (and
possibly requests too).
For example; I have 5 node cluster (replication factor of 3); and in which
all 4 has same size; where as one node is almost double in size for /hbase.
Any help will be appreciated; esp what makes the skew and how to fix it ?
Thanks
David
Re: Unbalanced region server
Posted by Bryan Beaudreault <bb...@hubspot.com>.
Unfortunately I don't have any experience with that version. You might try an hdfs rebalance, but I'm not sure of the implications of that for hbase. Hopefully someone else can be of more help. Good luck!
Sent from iPhone.
On Apr 24, 2012, at 12:07 AM, David Charle <db...@gmail.com> wrote:
> On Mon, Apr 23, 2012 at 8:43 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
>> wrote:
>
>> Over time the balancer will move around both existing and new regions. A
>> few questions: How many tables do you have? Are they all configured
>> similarly? What version of hbase are you on?
>>
>>
> 70 tables.
>
> hdfs://cluster1.com/hbase 2422000870355
>
> Yes; all configured similarly; and have uniform conf across all regions.
>
> 0.20.3 :) don't ask me to upgrade first, and then come back.. :(
>
> balance_switch is not supported in this version.
>
> Is there any other way to split and reclaim the space ?
>
> Thanks for the help
> David
>
>
>> Sent from iPhone.
>>
>> On Apr 23, 2012, at 11:39 PM, David Charle <db...@gmail.com> wrote:
>>
>>> Does balancer_switch=true; rebalances any skew in the existing keys or it
>>> only does for all new regions ? The requests seems to be oK across all
>>> servers.
>>>
>>> On Mon, Apr 23, 2012 at 8:03 PM, David Charle <db...@gmail.com>
>> wrote:
>>>
>>>> Thanks Bryan.
>>>>
>>>> Here is the regions count:
>>>>
>>>> 1: 2616
>>>> 2: 2620
>>>> 3: 2623
>>>> 4: 2617
>>>> 5: 2617
>>>>
>>>> The skew is in Node 2 where we have the space issue (double the size of
>>>> other 4 nodes).
>>>>
>>>>
>>>> On Mon, Apr 23, 2012 at 7:50 PM, Bryan Beaudreault <
>>>> bbeaudreault@hubspot.com> wrote:
>>>>
>>>>> Do all nodes have the same number of regions? If not you may not have
>> the
>>>>> balancer on. You can turn it on using balancer_switch true in the
>> hbase
>>>>> shell. The balancer also doesn't run if there is a region stuck in
>>>>> transition. In this case if your data is growing rapidly I have seen
>>>>> regionservers become lopsided until I could clean up the stuck region
>> and
>>>>> re-enable the balancer.
>>>>>
>>>>> Finally, depending on your version of hbase, the balancer only tries to
>>>>> keep the number of regions similar across all regionservers. If you
>> have
>>>>> tables of different max region sizes I could imagine a case where one
>>>>> region server unluckily is hosting more regions from the table with
>> larger
>>>>> region sizes. This also might explain the imbalance in requests, if
>> one
>>>>> of
>>>>> your tables gets more traffic than other. The HMaster UI would be
>> helpful
>>>>> for determining the spread of regions per table across regionservers.
>>>>>
>>>>> On Mon, Apr 23, 2012 at 9:41 PM, David Charle <db...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> What can make a region server unbalanced when it comes to space ? (and
>>>>>> possibly requests too).
>>>>>>
>>>>>> For example; I have 5 node cluster (replication factor of 3); and in
>>>>> which
>>>>>> all 4 has same size; where as one node is almost double in size for
>>>>> /hbase.
>>>>>>
>>>>>> Any help will be appreciated; esp what makes the skew and how to fix
>> it
>>>>> ?
>>>>>>
>>>>>> Thanks
>>>>>> David
>>>>>>
>>>>>
>>>>
>>>>
>>
Re: Unbalanced region server
Posted by Stack <st...@duboce.net>.
On Mon, Apr 23, 2012 at 9:07 PM, David Charle <db...@gmail.com> wrote:
> Is there any other way to split and reclaim the space ?
>
One node has more hdfs blocks than others? Is it a problem? Are you
coming close to running out of disk space? If you look in namenode
log, what happens when it tries to rebalance blocks? You could run
the hdfs balancer to move blocks but that will mess w/ hbase locality.
You have too many regions on your regionservers. Can you add more
nodes? Do you need that many tables? Drop those you don't need.
And yes, you should upgrade (smile).
St.Ack
Re: Unbalanced region server
Posted by David Charle <db...@gmail.com>.
On Mon, Apr 23, 2012 at 8:43 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
> wrote:
> Over time the balancer will move around both existing and new regions. A
> few questions: How many tables do you have? Are they all configured
> similarly? What version of hbase are you on?
>
>
70 tables.
hdfs://cluster1.com/hbase 2422000870355
Yes; all configured similarly; and have uniform conf across all regions.
0.20.3 :) don't ask me to upgrade first, and then come back.. :(
balance_switch is not supported in this version.
Is there any other way to split and reclaim the space ?
Thanks for the help
David
> Sent from iPhone.
>
> On Apr 23, 2012, at 11:39 PM, David Charle <db...@gmail.com> wrote:
>
> > Does balancer_switch=true; rebalances any skew in the existing keys or it
> > only does for all new regions ? The requests seems to be oK across all
> > servers.
> >
> > On Mon, Apr 23, 2012 at 8:03 PM, David Charle <db...@gmail.com>
> wrote:
> >
> >> Thanks Bryan.
> >>
> >> Here is the regions count:
> >>
> >> 1: 2616
> >> 2: 2620
> >> 3: 2623
> >> 4: 2617
> >> 5: 2617
> >>
> >> The skew is in Node 2 where we have the space issue (double the size of
> >> other 4 nodes).
> >>
> >>
> >> On Mon, Apr 23, 2012 at 7:50 PM, Bryan Beaudreault <
> >> bbeaudreault@hubspot.com> wrote:
> >>
> >>> Do all nodes have the same number of regions? If not you may not have
> the
> >>> balancer on. You can turn it on using balancer_switch true in the
> hbase
> >>> shell. The balancer also doesn't run if there is a region stuck in
> >>> transition. In this case if your data is growing rapidly I have seen
> >>> regionservers become lopsided until I could clean up the stuck region
> and
> >>> re-enable the balancer.
> >>>
> >>> Finally, depending on your version of hbase, the balancer only tries to
> >>> keep the number of regions similar across all regionservers. If you
> have
> >>> tables of different max region sizes I could imagine a case where one
> >>> region server unluckily is hosting more regions from the table with
> larger
> >>> region sizes. This also might explain the imbalance in requests, if
> one
> >>> of
> >>> your tables gets more traffic than other. The HMaster UI would be
> helpful
> >>> for determining the spread of regions per table across regionservers.
> >>>
> >>> On Mon, Apr 23, 2012 at 9:41 PM, David Charle <db...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi
> >>>>
> >>>> What can make a region server unbalanced when it comes to space ? (and
> >>>> possibly requests too).
> >>>>
> >>>> For example; I have 5 node cluster (replication factor of 3); and in
> >>> which
> >>>> all 4 has same size; where as one node is almost double in size for
> >>> /hbase.
> >>>>
> >>>> Any help will be appreciated; esp what makes the skew and how to fix
> it
> >>> ?
> >>>>
> >>>> Thanks
> >>>> David
> >>>>
> >>>
> >>
> >>
>
Re: Unbalanced region server
Posted by Bryan Beaudreault <bb...@hubspot.com>.
Over time the balancer will move around both existing and new regions. A few questions: How many tables do you have? Are they all configured similarly? What version of hbase are you on?
Sent from iPhone.
On Apr 23, 2012, at 11:39 PM, David Charle <db...@gmail.com> wrote:
> Does balancer_switch=true; rebalances any skew in the existing keys or it
> only does for all new regions ? The requests seems to be oK across all
> servers.
>
> On Mon, Apr 23, 2012 at 8:03 PM, David Charle <db...@gmail.com> wrote:
>
>> Thanks Bryan.
>>
>> Here is the regions count:
>>
>> 1: 2616
>> 2: 2620
>> 3: 2623
>> 4: 2617
>> 5: 2617
>>
>> The skew is in Node 2 where we have the space issue (double the size of
>> other 4 nodes).
>>
>>
>> On Mon, Apr 23, 2012 at 7:50 PM, Bryan Beaudreault <
>> bbeaudreault@hubspot.com> wrote:
>>
>>> Do all nodes have the same number of regions? If not you may not have the
>>> balancer on. You can turn it on using balancer_switch true in the hbase
>>> shell. The balancer also doesn't run if there is a region stuck in
>>> transition. In this case if your data is growing rapidly I have seen
>>> regionservers become lopsided until I could clean up the stuck region and
>>> re-enable the balancer.
>>>
>>> Finally, depending on your version of hbase, the balancer only tries to
>>> keep the number of regions similar across all regionservers. If you have
>>> tables of different max region sizes I could imagine a case where one
>>> region server unluckily is hosting more regions from the table with larger
>>> region sizes. This also might explain the imbalance in requests, if one
>>> of
>>> your tables gets more traffic than other. The HMaster UI would be helpful
>>> for determining the spread of regions per table across regionservers.
>>>
>>> On Mon, Apr 23, 2012 at 9:41 PM, David Charle <db...@gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> What can make a region server unbalanced when it comes to space ? (and
>>>> possibly requests too).
>>>>
>>>> For example; I have 5 node cluster (replication factor of 3); and in
>>> which
>>>> all 4 has same size; where as one node is almost double in size for
>>> /hbase.
>>>>
>>>> Any help will be appreciated; esp what makes the skew and how to fix it
>>> ?
>>>>
>>>> Thanks
>>>> David
>>>>
>>>
>>
>>
Re: Unbalanced region server
Posted by David Charle <db...@gmail.com>.
Does balancer_switch=true; rebalances any skew in the existing keys or it
only does for all new regions ? The requests seems to be oK across all
servers.
On Mon, Apr 23, 2012 at 8:03 PM, David Charle <db...@gmail.com> wrote:
> Thanks Bryan.
>
> Here is the regions count:
>
> 1: 2616
> 2: 2620
> 3: 2623
> 4: 2617
> 5: 2617
>
> The skew is in Node 2 where we have the space issue (double the size of
> other 4 nodes).
>
>
> On Mon, Apr 23, 2012 at 7:50 PM, Bryan Beaudreault <
> bbeaudreault@hubspot.com> wrote:
>
>> Do all nodes have the same number of regions? If not you may not have the
>> balancer on. You can turn it on using balancer_switch true in the hbase
>> shell. The balancer also doesn't run if there is a region stuck in
>> transition. In this case if your data is growing rapidly I have seen
>> regionservers become lopsided until I could clean up the stuck region and
>> re-enable the balancer.
>>
>> Finally, depending on your version of hbase, the balancer only tries to
>> keep the number of regions similar across all regionservers. If you have
>> tables of different max region sizes I could imagine a case where one
>> region server unluckily is hosting more regions from the table with larger
>> region sizes. This also might explain the imbalance in requests, if one
>> of
>> your tables gets more traffic than other. The HMaster UI would be helpful
>> for determining the spread of regions per table across regionservers.
>>
>> On Mon, Apr 23, 2012 at 9:41 PM, David Charle <db...@gmail.com>
>> wrote:
>>
>> > Hi
>> >
>> > What can make a region server unbalanced when it comes to space ? (and
>> > possibly requests too).
>> >
>> > For example; I have 5 node cluster (replication factor of 3); and in
>> which
>> > all 4 has same size; where as one node is almost double in size for
>> /hbase.
>> >
>> > Any help will be appreciated; esp what makes the skew and how to fix it
>> ?
>> >
>> > Thanks
>> > David
>> >
>>
>
>
Re: Unbalanced region server
Posted by David Charle <db...@gmail.com>.
Thanks Bryan.
Here is the regions count:
1: 2616
2: 2620
3: 2623
4: 2617
5: 2617
The skew is in Node 2 where we have the space issue (double the size of
other 4 nodes).
On Mon, Apr 23, 2012 at 7:50 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
> wrote:
> Do all nodes have the same number of regions? If not you may not have the
> balancer on. You can turn it on using balancer_switch true in the hbase
> shell. The balancer also doesn't run if there is a region stuck in
> transition. In this case if your data is growing rapidly I have seen
> regionservers become lopsided until I could clean up the stuck region and
> re-enable the balancer.
>
> Finally, depending on your version of hbase, the balancer only tries to
> keep the number of regions similar across all regionservers. If you have
> tables of different max region sizes I could imagine a case where one
> region server unluckily is hosting more regions from the table with larger
> region sizes. This also might explain the imbalance in requests, if one of
> your tables gets more traffic than other. The HMaster UI would be helpful
> for determining the spread of regions per table across regionservers.
>
> On Mon, Apr 23, 2012 at 9:41 PM, David Charle <db...@gmail.com>
> wrote:
>
> > Hi
> >
> > What can make a region server unbalanced when it comes to space ? (and
> > possibly requests too).
> >
> > For example; I have 5 node cluster (replication factor of 3); and in
> which
> > all 4 has same size; where as one node is almost double in size for
> /hbase.
> >
> > Any help will be appreciated; esp what makes the skew and how to fix it ?
> >
> > Thanks
> > David
> >
>
Re: Unbalanced region server
Posted by Bryan Beaudreault <bb...@hubspot.com>.
Do all nodes have the same number of regions? If not you may not have the
balancer on. You can turn it on using balancer_switch true in the hbase
shell. The balancer also doesn't run if there is a region stuck in
transition. In this case if your data is growing rapidly I have seen
regionservers become lopsided until I could clean up the stuck region and
re-enable the balancer.
Finally, depending on your version of hbase, the balancer only tries to
keep the number of regions similar across all regionservers. If you have
tables of different max region sizes I could imagine a case where one
region server unluckily is hosting more regions from the table with larger
region sizes. This also might explain the imbalance in requests, if one of
your tables gets more traffic than other. The HMaster UI would be helpful
for determining the spread of regions per table across regionservers.
On Mon, Apr 23, 2012 at 9:41 PM, David Charle <db...@gmail.com> wrote:
> Hi
>
> What can make a region server unbalanced when it comes to space ? (and
> possibly requests too).
>
> For example; I have 5 node cluster (replication factor of 3); and in which
> all 4 has same size; where as one node is almost double in size for /hbase.
>
> Any help will be appreciated; esp what makes the skew and how to fix it ?
>
> Thanks
> David
>