You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Saad Mufti <sa...@gmail.com> on 2018/03/21 00:18:29 UTC
Balance Regions Faster
Hi,
We are using the stochastic load balancer, and have tuned it to do a
maximum of 1% of regions in any calculation. But it is way too conservative
after that, it moves one region at a time. Is there a way to tell it to go
faster with whatever number of regions it decided to do? I have been
looking at the settings in the code but so far a little confused about
which exact setting will achieve this. Is it one of the steps settings?
Also our cluster was in bad shape for a while as a bunch of region servers
aborted for different reasons. When we stopped everything and brought it
back up and enabled the tables, most of the region servers were assigned no
regions. Is it for locality reasons that HBase is trying to assign regions
where they were assigned before? Is there a way to tell HBase to ignore
that on startup?
Thanks.
----
Saad
Re: Balance Regions Faster
Posted by Saad Mufti <sa...@gmail.com>.
Thanks, will take a look. We're using HBase 1.4.0 on AWS EMR.
Cheers.
----
Saad
On Tue, Mar 20, 2018 at 9:32 PM, Ted Yu <yu...@gmail.com> wrote:
> Saad:
> You didn't mention the version of hbase you are using.
> Please check the version to see if the following were included:
>
> HBASE-18164 Fast locality computation in balancer
> HBASE-16570 Compute region locality in parallel at startup
> HBASE-15515 Improve LocalityBasedCandidateGenerator in Balancer
>
> Cheers
>
> On Tue, Mar 20, 2018 at 6:11 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Please consider tuning the following parameters of stochastic load
> > balancer :
> >
> > "hbase.master.balancer.stochastic.maxRunningTime"
> >
> > default value is 30 seconds. It controls the duration of runtime for
> each balanceCluster()
> > call.
> >
> > "hbase.balancer.period"
> >
> > default is 300 seconds. It controls the maximum time master runs balancer
> > for.
> >
> > You can turn on DEBUG logging and observe the following output in master
> > log:
> >
> > balancer.StochasticLoadBalancer: Finished computing new load balance
> plan. Computation took 1200227ms to try 2254 different iterations. Found
> a solution that moves 550 regions; Going from a computed cost of
> 77.52829271038965 to a new cost of 74.32764924425548
> >
> > If you have a dev cluster, you can try different combinations of the
> above
> > two parameters and get best performance by checking the above log.
> >
> > Cheers
> >
> > On Tue, Mar 20, 2018 at 5:18 PM, Saad Mufti <sa...@gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> We are using the stochastic load balancer, and have tuned it to do a
> >> maximum of 1% of regions in any calculation. But it is way too
> >> conservative
> >> after that, it moves one region at a time. Is there a way to tell it to
> go
> >> faster with whatever number of regions it decided to do? I have been
> >> looking at the settings in the code but so far a little confused about
> >> which exact setting will achieve this. Is it one of the steps settings?
> >>
> >> Also our cluster was in bad shape for a while as a bunch of region
> servers
> >> aborted for different reasons. When we stopped everything and brought it
> >> back up and enabled the tables, most of the region servers were assigned
> >> no
> >> regions. Is it for locality reasons that HBase is trying to assign
> regions
> >> where they were assigned before? Is there a way to tell HBase to ignore
> >> that on startup?
> >>
> >> Thanks.
> >>
> >> ----
> >> Saad
> >>
> >
> >
>
Re: Balance Regions Faster
Posted by Ted Yu <yu...@gmail.com>.
Saad:
You didn't mention the version of hbase you are using.
Please check the version to see if the following were included:
HBASE-18164 Fast locality computation in balancer
HBASE-16570 Compute region locality in parallel at startup
HBASE-15515 Improve LocalityBasedCandidateGenerator in Balancer
Cheers
On Tue, Mar 20, 2018 at 6:11 PM, Ted Yu <yu...@gmail.com> wrote:
> Please consider tuning the following parameters of stochastic load
> balancer :
>
> "hbase.master.balancer.stochastic.maxRunningTime"
>
> default value is 30 seconds. It controls the duration of runtime for each balanceCluster()
> call.
>
> "hbase.balancer.period"
>
> default is 300 seconds. It controls the maximum time master runs balancer
> for.
>
> You can turn on DEBUG logging and observe the following output in master
> log:
>
> balancer.StochasticLoadBalancer: Finished computing new load balance plan. Computation took 1200227ms to try 2254 different iterations. Found a solution that moves 550 regions; Going from a computed cost of 77.52829271038965 to a new cost of 74.32764924425548
>
> If you have a dev cluster, you can try different combinations of the above
> two parameters and get best performance by checking the above log.
>
> Cheers
>
> On Tue, Mar 20, 2018 at 5:18 PM, Saad Mufti <sa...@gmail.com> wrote:
>
>> Hi,
>>
>> We are using the stochastic load balancer, and have tuned it to do a
>> maximum of 1% of regions in any calculation. But it is way too
>> conservative
>> after that, it moves one region at a time. Is there a way to tell it to go
>> faster with whatever number of regions it decided to do? I have been
>> looking at the settings in the code but so far a little confused about
>> which exact setting will achieve this. Is it one of the steps settings?
>>
>> Also our cluster was in bad shape for a while as a bunch of region servers
>> aborted for different reasons. When we stopped everything and brought it
>> back up and enabled the tables, most of the region servers were assigned
>> no
>> regions. Is it for locality reasons that HBase is trying to assign regions
>> where they were assigned before? Is there a way to tell HBase to ignore
>> that on startup?
>>
>> Thanks.
>>
>> ----
>> Saad
>>
>
>
Re: Balance Regions Faster
Posted by Ted Yu <yu...@gmail.com>.
Please consider tuning the following parameters of stochastic load balancer
:
"hbase.master.balancer.stochastic.maxRunningTime"
default value is 30 seconds. It controls the duration of runtime for
each balanceCluster()
call.
"hbase.balancer.period"
default is 300 seconds. It controls the maximum time master runs balancer
for.
You can turn on DEBUG logging and observe the following output in master
log:
balancer.StochasticLoadBalancer: Finished computing new load balance
plan. Computation took 1200227ms to try 2254 different iterations.
Found a solution that moves 550 regions; Going from a computed cost of
77.52829271038965 to a new cost of 74.32764924425548
If you have a dev cluster, you can try different combinations of the above
two parameters and get best performance by checking the above log.
Cheers
On Tue, Mar 20, 2018 at 5:18 PM, Saad Mufti <sa...@gmail.com> wrote:
> Hi,
>
> We are using the stochastic load balancer, and have tuned it to do a
> maximum of 1% of regions in any calculation. But it is way too conservative
> after that, it moves one region at a time. Is there a way to tell it to go
> faster with whatever number of regions it decided to do? I have been
> looking at the settings in the code but so far a little confused about
> which exact setting will achieve this. Is it one of the steps settings?
>
> Also our cluster was in bad shape for a while as a bunch of region servers
> aborted for different reasons. When we stopped everything and brought it
> back up and enabled the tables, most of the region servers were assigned no
> regions. Is it for locality reasons that HBase is trying to assign regions
> where they were assigned before? Is there a way to tell HBase to ignore
> that on startup?
>
> Thanks.
>
> ----
> Saad
>