You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by Дмитрий Павлов <dm...@inbox.ru> on 2018/12/03 12:10:55 UTC
[KUDU] Rebalancing tool
Hi guys
I have a question about Kudu rebalancing tool ( kudu cluster rebalance ).
So my situation is following:
When our kudu cluster has been updated to version 1.8.0 i ran kudu cluster rebalance for one table.
After that i checked per-table replica distribution and it looks fine but leaders distributions per nodes still not good for my table
Is it expected behaviour for rebalancing tool?
Regards, Dmitry Pavlov
Re: Re[2]: [KUDU] Rebalancing tool
Posted by William Berkeley <wd...@cloudera.com>.
Yeah, it's worth looking into. If leadership tends to migrate to a subset
of the servers it indicates some kind of instability or communication
problems with the other servers, since leadership will move away from
servers that have trouble sending timely heartbeats to followers.
The leader does do more work for a tablet than the followers, and writes
must go into the leader's log as well as the log of a majority, so there's
some penalty on write latency and throughput to have leadership
concentrated.
The next version of Kudu will have an enhanced leader_step_down tool that
implements KUDU-2245, so leadership can be transferred with minimal
disruption to the tablet and it can be transferred to a specific replica
(as long as that replica is almost caught up to the leader).
-Will
On Tue, Dec 4, 2018 at 5:04 AM Дмитрий Павлов <dm...@inbox.ru> wrote:
> Thank Will
>
> So i have one more question.
>
> is this (i mean leaders skew) something i should be concerned about?
> In terms of load balancing for example in case if i use kudu 1.8.0 with
> spark
>
> Regards Dmitry Pavlov
>
>
> Вторник, 4 декабря 2018, 0:10 +03:00 от William Berkeley <
> wdberkeley@cloudera.com>:
>
> Yes, it's expected. The rebalancing tool does not balance leadership. In
> fact, it tries to avoid relocating leader replicas because the tool wants
> to minimize the disturbance to the cluster. It will relocate leader
> replicas if it has to, but it won't make any attempt to balance them.
>
> You can use 'kudu tablet leader_step_down' to force the leader replica of
> a tablet to step down. A new leader will be elected after the election
> timeout passes. It's not guaranteed that the old leader won't be reelected.
>
> -Will
>
> On Mon, Dec 3, 2018 at 4:11 AM Дмитрий Павлов <dm.pavlov@inbox.ru
> <ht...@inbox.ru>> wrote:
>
>
> Hi guys
>
> I have a question about Kudu rebalancing tool (*kudu cluster rebalance*).
>
> So my situation is following:
>
> When our kudu cluster has been updated to version 1.8.0 i ran *kudu
> cluster rebalance *for one table.
> After that i checked per-table replica distribution and it looks fine but
> leaders distributions per nodes still not good for my table
>
> Is it expected behaviour for rebalancing tool?
>
> Regards, Dmitry Pavlov
>
>
>
>
>
> --
> Дмитрий Павлов
>
Re[2]: [KUDU] Rebalancing tool
Posted by Дмитрий Павлов <dm...@inbox.ru>.
Thank Will
So i have one more question.
is this (i mean leaders skew) something i should be concerned about?
In terms of load balancing for example in case if i use kudu 1.8.0 with spark
Regards Dmitry Pavlov
>Вторник, 4 декабря 2018, 0:10 +03:00 от William Berkeley <wd...@cloudera.com>:
>
>Yes, it's expected. The rebalancing tool does not balance leadership. In fact, it tries to avoid relocating leader replicas because the tool wants to minimize the disturbance to the cluster. It will relocate leader replicas if it has to, but it won't make any attempt to balance them.
>
>You can use 'kudu tablet leader_step_down' to force the leader replica of a tablet to step down. A new leader will be elected after the election timeout passes. It's not guaranteed that the old leader won't be reelected.
>
>-Will
>On Mon, Dec 3, 2018 at 4:11 AM Дмитрий Павлов < dm.pavlov@inbox.ru > wrote:
>>
>>Hi guys
>>
>>I have a question about Kudu rebalancing tool ( kudu cluster rebalance ).
>>
>>So my situation is following:
>>
>>When our kudu cluster has been updated to version 1.8.0 i ran kudu cluster rebalance for one table.
>>After that i checked per-table replica distribution and it looks fine but leaders distributions per nodes still not good for my table
>>
>>Is it expected behaviour for rebalancing tool?
>>
>>Regards, Dmitry Pavlov
>>
>>
>>
--
Дмитрий Павлов
Re: [KUDU] Rebalancing tool
Posted by William Berkeley <wd...@cloudera.com>.
Yes, it's expected. The rebalancing tool does not balance leadership. In
fact, it tries to avoid relocating leader replicas because the tool wants
to minimize the disturbance to the cluster. It will relocate leader
replicas if it has to, but it won't make any attempt to balance them.
You can use 'kudu tablet leader_step_down' to force the leader replica of a
tablet to step down. A new leader will be elected after the election
timeout passes. It's not guaranteed that the old leader won't be reelected.
-Will
On Mon, Dec 3, 2018 at 4:11 AM Дмитрий Павлов <dm...@inbox.ru> wrote:
>
> Hi guys
>
> I have a question about Kudu rebalancing tool (*kudu cluster rebalance*).
>
> So my situation is following:
>
> When our kudu cluster has been updated to version 1.8.0 i ran *kudu
> cluster rebalance *for one table.
> After that i checked per-table replica distribution and it looks fine but
> leaders distributions per nodes still not good for my table
>
> Is it expected behaviour for rebalancing tool?
>
> Regards, Dmitry Pavlov
>
>
>
>