You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Meghana Narasimhan <mn...@bandwidth.com> on 2017/01/18 14:46:01 UTC
Leader imbalance issue
Hi,
We have a 3 node cluster with 0.9.0.1 version. The controller is reporting
that one of the brokers has a leader imbalance.
One of the topics with 120 partitions has Broker 0 and 2 acting as leaders
for all its partitions.None of the partitions have Broker 1 as their leader.
So the controller log reports a list of partitions for those topics not
being in the preferred
replica Map and also leader imbalance.
DEBUG [Controller 1]: topics not in preferred replica Map([test,48] ->
List(1, 2, 0), [test,54] -> List(1, 2, 0)...............................)
TRACE [Controller 1]: leader imbalance ratio for broker 1 is 0.086957
(kafka.controller.KafkaController)
I believe broker 1 got into this state because of an issue on that broker a
few weeks backup
which crashed Kafka on that node. Later the corrupted log was identified
and restored.
But looks like since then that Broker hasn't been leader for that topic.
What is the correct way to fix the issue of leader imbalance ?
Thanks,
Meghana
Re: Leader imbalance issue
Posted by Meghana Narasimhan <mn...@bandwidth.com>.
Thanks Apurva ! Will give that a shot.
Thanks,
Meghana
On Fri, Jan 20, 2017 at 2:16 PM, Apurva Mehta <ap...@confluent.io> wrote:
> Hi Meghana,
>
> Have you tried using the 'kafka-prefered-replica-election.sh' script? It
> will try to move leaders back to the preferred replicas when there is a
> leader imbalance.
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#
> Replicationtools-1.PreferredReplicaLeaderElectionTool
>
> Thanks,
> Apurva
>
>
> On Fri, Jan 20, 2017 at 8:29 AM, Meghana Narasimhan <
> mnarasimhan@bandwidth.com> wrote:
>
> > Hi,
> > Any input on this will be of great help . A rolling restart could fix the
> > issue but not sure if thats the right way to do it.
> >
> > Thanks,
> > Meghana
> >
> > On Wed, Jan 18, 2017 at 9:46 AM, Meghana Narasimhan <
> > mnarasimhan@bandwidth.com> wrote:
> >
> > > Hi,
> > >
> > > We have a 3 node cluster with 0.9.0.1 version. The controller is
> > reporting
> > > that one of the brokers has a leader imbalance.
> > > One of the topics with 120 partitions has Broker 0 and 2 acting as
> > leaders
> > > for all its partitions.None of the partitions have Broker 1 as their
> > leader.
> > >
> > > So the controller log reports a list of partitions for those topics
> not
> > > being in the preferred
> > > replica Map and also leader imbalance.
> > >
> > > DEBUG [Controller 1]: topics not in preferred replica Map([test,48] ->
> > > List(1, 2, 0), [test,54] -> List(1, 2, 0)............................
> > ...)
> > > TRACE [Controller 1]: leader imbalance ratio for broker 1 is 0.086957
> > > (kafka.controller.KafkaController)
> > >
> > > I believe broker 1 got into this state because of an issue on that
> broker
> > > a few weeks backup
> > > which crashed Kafka on that node. Later the corrupted log was
> identified
> > > and restored.
> > > But looks like since then that Broker hasn't been leader for that
> topic.
> > >
> > > What is the correct way to fix the issue of leader imbalance ?
> > >
> > > Thanks,
> > > Meghana
> > >
> > >
> > >
> >
>
Re: Leader imbalance issue
Posted by Apurva Mehta <ap...@confluent.io>.
Hi Meghana,
Have you tried using the 'kafka-prefered-replica-election.sh' script? It
will try to move leaders back to the preferred replicas when there is a
leader imbalance.
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.PreferredReplicaLeaderElectionTool
Thanks,
Apurva
On Fri, Jan 20, 2017 at 8:29 AM, Meghana Narasimhan <
mnarasimhan@bandwidth.com> wrote:
> Hi,
> Any input on this will be of great help . A rolling restart could fix the
> issue but not sure if thats the right way to do it.
>
> Thanks,
> Meghana
>
> On Wed, Jan 18, 2017 at 9:46 AM, Meghana Narasimhan <
> mnarasimhan@bandwidth.com> wrote:
>
> > Hi,
> >
> > We have a 3 node cluster with 0.9.0.1 version. The controller is
> reporting
> > that one of the brokers has a leader imbalance.
> > One of the topics with 120 partitions has Broker 0 and 2 acting as
> leaders
> > for all its partitions.None of the partitions have Broker 1 as their
> leader.
> >
> > So the controller log reports a list of partitions for those topics not
> > being in the preferred
> > replica Map and also leader imbalance.
> >
> > DEBUG [Controller 1]: topics not in preferred replica Map([test,48] ->
> > List(1, 2, 0), [test,54] -> List(1, 2, 0)............................
> ...)
> > TRACE [Controller 1]: leader imbalance ratio for broker 1 is 0.086957
> > (kafka.controller.KafkaController)
> >
> > I believe broker 1 got into this state because of an issue on that broker
> > a few weeks backup
> > which crashed Kafka on that node. Later the corrupted log was identified
> > and restored.
> > But looks like since then that Broker hasn't been leader for that topic.
> >
> > What is the correct way to fix the issue of leader imbalance ?
> >
> > Thanks,
> > Meghana
> >
> >
> >
>
Re: Leader imbalance issue
Posted by Meghana Narasimhan <mn...@bandwidth.com>.
Hi,
Any input on this will be of great help . A rolling restart could fix the
issue but not sure if thats the right way to do it.
Thanks,
Meghana
On Wed, Jan 18, 2017 at 9:46 AM, Meghana Narasimhan <
mnarasimhan@bandwidth.com> wrote:
> Hi,
>
> We have a 3 node cluster with 0.9.0.1 version. The controller is reporting
> that one of the brokers has a leader imbalance.
> One of the topics with 120 partitions has Broker 0 and 2 acting as leaders
> for all its partitions.None of the partitions have Broker 1 as their leader.
>
> So the controller log reports a list of partitions for those topics not
> being in the preferred
> replica Map and also leader imbalance.
>
> DEBUG [Controller 1]: topics not in preferred replica Map([test,48] ->
> List(1, 2, 0), [test,54] -> List(1, 2, 0)...............................)
> TRACE [Controller 1]: leader imbalance ratio for broker 1 is 0.086957
> (kafka.controller.KafkaController)
>
> I believe broker 1 got into this state because of an issue on that broker
> a few weeks backup
> which crashed Kafka on that node. Later the corrupted log was identified
> and restored.
> But looks like since then that Broker hasn't been leader for that topic.
>
> What is the correct way to fix the issue of leader imbalance ?
>
> Thanks,
> Meghana
>
>
>