You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kane Kim <ka...@gmail.com> on 2016/04/26 19:33:15 UTC
auto leader rebalancing
Hello,
We have auto.leader.rebalance.enable = True, other options are by default
(10% imbalance ratio and 300 seconds).
We have a check that reports leadership imbalance:
critical: Leadership out of balance for topic mp-auth. Leader counts: {
"104224873"=>84, "104224876"=>22, "104224877"=>55, "104224863"=>110,
"104224865"=>107, "104224867"=>105, "104224868"=>42, "104224871"=>9,
"104224879"=>20, "104224866"=>12, "104224869"=>17, "104224864"=>14,
"104224878"=>3}
Above is the brokerid: number of partitions, apparently the imbalance is
higher than 10% and it was in this state for a couple of hours, there is no
evidence that it's trying to rebalance.
Is there anything else we have to configure/check?
Thanks.
Re: auto leader rebalancing
Posted by Kane Kim <ka...@gmail.com>.
No, I don't see that in controller.log. Looks like it's related to this
bug: https://issues.apache.org/jira/browse/KAFKA-2729, after I did rolling
restart (all brokers) it rebalanced. It wasn't rebalancing though when a
couple of brokers were missing from ISR (but there were enough brokers to
rebalance anyway).
On Wed, Apr 27, 2016 at 4:01 PM, Robert Christ <rc...@tivo.com> wrote:
> I believe the 10% is measured on a broker level not a topic
> level. Do you see lines like:
>
> [2016-04-27 22:52:47,854] TRACE [Controller 3]: leader imbalance ratio for
> broker 5 is 0.978555 (kafka.controller.KafkaController)
> [2016-04-27 22:52:47,855] TRACE [Controller 3]: leader imbalance ratio for
> broker 1 is 0.000000 (kafka.controller.KafkaController)
> [2016-04-27 22:52:47,855] TRACE [Controller 3]: leader imbalance ratio for
> broker 2 is 0.000000 (kafka.controller.KafkaController)
> [2016-04-27 22:52:47,855] TRACE [Controller 3]: leader imbalance ratio for
> broker 3 is 0.000000 (kafka.controller.KafkaController)
> [2016-04-27 22:52:47,855] TRACE [Controller 3]: leader imbalance ratio for
> broker 4 is 0.000000 (kafka.controller.KafkaController)
>
> in your controller log?
> rob
>
> > On Apr 27, 2016, at 3:46 PM, Kane Kim <ka...@gmail.com> wrote:
> >
> > Bump
> >
> > On Tue, Apr 26, 2016 at 10:33 AM, Kane Kim <ka...@gmail.com>
> wrote:
> >
> >> Hello,
> >>
> >> We have auto.leader.rebalance.enable = True, other options are by
> default
> >> (10% imbalance ratio and 300 seconds).
> >>
> >> We have a check that reports leadership imbalance:
> >>
> >> critical: Leadership out of balance for topic mp-auth. Leader counts: {
> >> "104224873"=>84, "104224876"=>22, "104224877"=>55, "104224863"=>110,
> >> "104224865"=>107, "104224867"=>105, "104224868"=>42, "104224871"=>9,
> >> "104224879"=>20, "104224866"=>12, "104224869"=>17, "104224864"=>14,
> >> "104224878"=>3}
> >>
> >> Above is the brokerid: number of partitions, apparently the imbalance is
> >> higher than 10% and it was in this state for a couple of hours, there
> is no
> >> evidence that it's trying to rebalance.
> >>
> >> Is there anything else we have to configure/check?
> >>
> >> Thanks.
> >>
>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>
Re: auto leader rebalancing
Posted by Robert Christ <rc...@tivo.com>.
I believe the 10% is measured on a broker level not a topic
level. Do you see lines like:
[2016-04-27 22:52:47,854] TRACE [Controller 3]: leader imbalance ratio for broker 5 is 0.978555 (kafka.controller.KafkaController)
[2016-04-27 22:52:47,855] TRACE [Controller 3]: leader imbalance ratio for broker 1 is 0.000000 (kafka.controller.KafkaController)
[2016-04-27 22:52:47,855] TRACE [Controller 3]: leader imbalance ratio for broker 2 is 0.000000 (kafka.controller.KafkaController)
[2016-04-27 22:52:47,855] TRACE [Controller 3]: leader imbalance ratio for broker 3 is 0.000000 (kafka.controller.KafkaController)
[2016-04-27 22:52:47,855] TRACE [Controller 3]: leader imbalance ratio for broker 4 is 0.000000 (kafka.controller.KafkaController)
in your controller log?
rob
> On Apr 27, 2016, at 3:46 PM, Kane Kim <ka...@gmail.com> wrote:
>
> Bump
>
> On Tue, Apr 26, 2016 at 10:33 AM, Kane Kim <ka...@gmail.com> wrote:
>
>> Hello,
>>
>> We have auto.leader.rebalance.enable = True, other options are by default
>> (10% imbalance ratio and 300 seconds).
>>
>> We have a check that reports leadership imbalance:
>>
>> critical: Leadership out of balance for topic mp-auth. Leader counts: {
>> "104224873"=>84, "104224876"=>22, "104224877"=>55, "104224863"=>110,
>> "104224865"=>107, "104224867"=>105, "104224868"=>42, "104224871"=>9,
>> "104224879"=>20, "104224866"=>12, "104224869"=>17, "104224864"=>14,
>> "104224878"=>3}
>>
>> Above is the brokerid: number of partitions, apparently the imbalance is
>> higher than 10% and it was in this state for a couple of hours, there is no
>> evidence that it's trying to rebalance.
>>
>> Is there anything else we have to configure/check?
>>
>> Thanks.
>>
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
Re: auto leader rebalancing
Posted by Kane Kim <ka...@gmail.com>.
Bump
On Tue, Apr 26, 2016 at 10:33 AM, Kane Kim <ka...@gmail.com> wrote:
> Hello,
>
> We have auto.leader.rebalance.enable = True, other options are by default
> (10% imbalance ratio and 300 seconds).
>
> We have a check that reports leadership imbalance:
>
> critical: Leadership out of balance for topic mp-auth. Leader counts: {
> "104224873"=>84, "104224876"=>22, "104224877"=>55, "104224863"=>110,
> "104224865"=>107, "104224867"=>105, "104224868"=>42, "104224871"=>9,
> "104224879"=>20, "104224866"=>12, "104224869"=>17, "104224864"=>14,
> "104224878"=>3}
>
> Above is the brokerid: number of partitions, apparently the imbalance is
> higher than 10% and it was in this state for a couple of hours, there is no
> evidence that it's trying to rebalance.
>
> Is there anything else we have to configure/check?
>
> Thanks.
>