You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Bongyeon Kim <bo...@gmail.com> on 2014/06/17 03:05:38 UTC
ISR not updating
Hi, team.
Im using Kafka 0.8.1.1.
I'm running 8 brokers on 4 machine. (2 brokers on 1 machine) and I have 3
topics each have 16 partitions and 3 replicas.
kafka-topics describe is
Topic:topicCDR PartitionCount:16 ReplicationFactor:3 Configs:retention.ms
=3600000
Topic: topicCDR Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,2
Topic: topicCDR Partition: 1 Leader: 4 Replicas: 4,2,3 Isr: 3,4,2
Topic: topicCDR Partition: 2 Leader: 5 Replicas: 5,3,4 Isr: 3,4,5
Topic: topicCDR Partition: 3 Leader: 6 Replicas: 6,4,5 Isr: 4,5,6
Topic: topicCDR Partition: 4 Leader: 7 Replicas: 7,5,6 Isr: 5,6,7
Topic: topicCDR Partition: 5 Leader: 8 Replicas: 8,6,7 Isr: 6,7,8
Topic: topicCDR Partition: 6 Leader: 1 Replicas: 1,7,8 Isr: 1,7,8
Topic: topicCDR Partition: 7 Leader: 2 Replicas: 2,8,1 Isr: 8,2
Topic: topicCDR Partition: 8 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
Topic: topicCDR Partition: 9 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
Topic: topicCDR Partition: 10 Leader: 5 Replicas: 5,4,6 Isr: 4,5,6
Topic: topicCDR Partition: 11 Leader: 6 Replicas: 6,5,7 Isr: 5,6,7
Topic: topicCDR Partition: 12 Leader: 7 Replicas: 7,6,8 Isr: 6,7,8
Topic: topicCDR Partition: 13 Leader: 8 Replicas: 8,7,1 Isr: 7,8
Topic: topicCDR Partition: 14 Leader: 8 Replicas: 1,8,2 Isr: 8,2
Topic: topicCDR Partition: 15 Leader: 2 Replicas: 2,1,3 Isr: 3,2
Topic:topicDEBUG PartitionCount:16 ReplicationFactor:3 Configs:retention.ms
=3600000
Topic: topicDEBUG Partition: 0 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
Topic: topicDEBUG Partition: 1 Leader: 5 Replicas: 5,4,6 Isr: 4,5,6
Topic: topicDEBUG Partition: 2 Leader: 6 Replicas: 6,5,7 Isr: 5,6,7
Topic: topicDEBUG Partition: 3 Leader: 7 Replicas: 7,6,8 Isr: 6,7,8
Topic: topicDEBUG Partition: 4 Leader: 8 Replicas: 8,7,1 Isr: 7,8
Topic: topicDEBUG Partition: 5 Leader: 8 Replicas: 1,8,2 Isr: 8,2
Topic: topicDEBUG Partition: 6 Leader: 2 Replicas: 2,1,3 Isr: 3,2
Topic: topicDEBUG Partition: 7 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
Topic: topicDEBUG Partition: 8 Leader: 4 Replicas: 4,5,6 Isr: 4,5,6
Topic: topicDEBUG Partition: 9 Leader: 5 Replicas: 5,6,7 Isr: 5,6,7
Topic: topicDEBUG Partition: 10 Leader: 6 Replicas: 6,7,8 Isr: 6,7,8
Topic: topicDEBUG Partition: 11 Leader: 7 Replicas: 7,8,1 Isr: 7,8,1
Topic: topicDEBUG Partition: 12 Leader: 8 Replicas: 8,1,2 Isr: 8,2
Topic: topicDEBUG Partition: 13 Leader: 3 Replicas: 1,2,3 Isr: 3,2
Topic: topicDEBUG Partition: 14 Leader: 2 Replicas: 2,3,4 Isr: 3,4,2
Topic: topicDEBUG Partition: 15 Leader: 3 Replicas: 3,4,5 Isr: 3,4,5
Topic:topicTRACE PartitionCount:16 ReplicationFactor:3 Configs:retention.ms
=3600000
Topic: topicTRACE Partition: 0 Leader: 5 Replicas: 5,8,1 Isr: 5,8,1
Topic: topicTRACE Partition: 1 Leader: 6 Replicas: 6,1,2 Isr: 6,1,2
Topic: topicTRACE Partition: 2 Leader: 7 Replicas: 7,2,3 Isr: 3,7,2
Topic: topicTRACE Partition: 3 Leader: 8 Replicas: 8,3,4 Isr: 3,4,8
Topic: topicTRACE Partition: 4 Leader: 1 Replicas: 1,4,5 Isr: 1,5,4
Topic: topicTRACE Partition: 5 Leader: 2 Replicas: 2,5,6 Isr: 5,6,2
Topic: topicTRACE Partition: 6 Leader: 3 Replicas: 3,6,7 Isr: 3,6,7
Topic: topicTRACE Partition: 7 Leader: 4 Replicas: 4,7,8 Isr: 4,7,8
Topic: topicTRACE Partition: 8 Leader: 5 Replicas: 5,1,2 Isr: 5,1,2
Topic: topicTRACE Partition: 9 Leader: 6 Replicas: 6,2,3 Isr: 3,6,2
Topic: topicTRACE Partition: 10 Leader: 7 Replicas: 7,3,4 Isr: 3,4,7
Topic: topicTRACE Partition: 11 Leader: 8 Replicas: 8,4,5 Isr: 4,5,8
Problem is one of my topic's ISR is not updating and keep failing to be
preferred replica. more detail, broker 1 for topicDEBUG's ISR is not
updating.
And log of broker 1 is absolutely normal and has no error.
This is expected situation? what I have to updating this?
Thanks in advance.
--
*Sincerely*
*,**Bongyeon Kim*
Java Developer & Engineer
Seoul, Korea
Mobile: +82-10-9369-1314
Email: bongyeonkim@gmail.com
Twitter: http://twitter.com/tigerby
Facebook: http://facebook.com/tigerby
Wiki: http://tigerby.com
Re: ISR not updating
Posted by Bongyeon Kim <bo...@gmail.com>.
I'm not enable controllerd.shutdown property. anyway, I'd better disable
auto leader banancing.
I would like to find that information on documentation.
Thanks. Jun
On Wed, Jun 18, 2014 at 11:47 PM, Jun Rao <ju...@gmail.com> wrote:
> Auto leader balancing has some know issues when using together with
> controlled shutdown. So, we don't recommend it to be turned on in 0.8.1.1
>
> Thanks,
>
> Jun
>
>
> On Wed, Jun 18, 2014 at 1:41 AM, Bongyeon Kim <bo...@gmail.com>
> wrote:
>
> > Yes.
> > it contain my server.properties file.
> >
> >
> > auto.leader.rebalance.enable=true
> >
> >
> > On Wed, Jun 18, 2014 at 12:44 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Did you have auto leader balancing enabled?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Jun 17, 2014 at 5:06 PM, Bongyeon Kim <bo...@gmail.com>
> > > wrote:
> > >
> > > > There is some error log about failing leader election like that.
> > > >
> > > >
> > > > [2014-06-18 08:59:21,014] ERROR Controller 7 epoch 4 encountered
> error
> > > > while electing leader for partition [topicDEBUG,5] due to: Preferred
> > > > replica 1 for partition [topicDEBUG,5] is either not alive or not in
> > the
> > > > isr. Current leader and ISR:
> > [{"leader":8,"leader_epoch":6,"isr":[8,2]}].
> > > > (state.change.logger)
> > > > [2014-06-18 08:59:21,014] ERROR Controller 7 epoch 4 initiated state
> > > change
> > > > for partition [topicDEBUG,5] from OnlinePartition to OnlinePartition
> > > failed
> > > > (state.change.logger)
> > > > kafka.common.StateChangeFailedException: encountered error while
> > electing
> > > > leader for partition [topicDEBUG,5] due to: Preferred replica 1 for
> > > > partition [topicDEBUG,5] is either not alive or not in the isr.
> Current
> > > > leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}].
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:360)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:187)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:125)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:124)
> > > > at scala.collection.immutable.Set$Set1.foreach(Set.scala:86)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:124)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.KafkaController.onPreferredReplicaElection(KafkaController.scala:618)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply$mcV$sp(KafkaController.scala:1118)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply(KafkaController.scala:1112)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply(KafkaController.scala:1112)
> > > > at kafka.utils.Utils$.inLock(Utils.scala:538)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1109)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1107)
> > > > at
> > > >
> > >
> >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
> > > > at
> > > >
> > >
> >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
> > > > at scala.collection.Iterator$class.foreach(Iterator.scala:772)
> > > > at
> > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
> > > > at
> > > >
> > >
> >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
> > > > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
> > > > at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1107)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1086)
> > > > at
> > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> > > > at
> > > >
> > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.KafkaController.kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance(KafkaController.scala:1086)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$onControllerFailover$1.apply$mcV$sp(KafkaController.scala:324)
> > > > at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100)
> > > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > at java.lang.Thread.run(Thread.java:744)
> > > > Caused by: kafka.common.StateChangeFailedException: Preferred
> replica 1
> > > for
> > > > partition [topicDEBUG,5] is either not alive or not in the isr.
> Current
> > > > leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}]
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.PreferredReplicaPartitionLeaderSelector.selectLeader(PartitionLeaderSelector.scala:144)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:336)
> > > > ... 33 more
> > > >
> > > >
> > > >
> > > > Thanks.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Jun 17, 2014 at 12:42 PM, Jun Rao <ju...@gmail.com> wrote:
> > > >
> > > > > Any error in the controller and state-change log?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, Jun 16, 2014 at 6:05 PM, Bongyeon Kim <
> > bongyeon.kim@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi, team.
> > > > > >
> > > > > > Im using Kafka 0.8.1.1.
> > > > > > I'm running 8 brokers on 4 machine. (2 brokers on 1 machine) and
> I
> > > > have 3
> > > > > > topics each have 16 partitions and 3 replicas.
> > > > > >
> > > > > > kafka-topics describe is
> > > > > >
> > > > > > Topic:topicCDR PartitionCount:16 ReplicationFactor:3 Configs:
> > > > > retention.ms
> > > > > > =3600000
> > > > > > Topic: topicCDR Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,2
> > > > > > Topic: topicCDR Partition: 1 Leader: 4 Replicas: 4,2,3 Isr: 3,4,2
> > > > > > Topic: topicCDR Partition: 2 Leader: 5 Replicas: 5,3,4 Isr: 3,4,5
> > > > > > Topic: topicCDR Partition: 3 Leader: 6 Replicas: 6,4,5 Isr: 4,5,6
> > > > > > Topic: topicCDR Partition: 4 Leader: 7 Replicas: 7,5,6 Isr: 5,6,7
> > > > > > Topic: topicCDR Partition: 5 Leader: 8 Replicas: 8,6,7 Isr: 6,7,8
> > > > > > Topic: topicCDR Partition: 6 Leader: 1 Replicas: 1,7,8 Isr: 1,7,8
> > > > > > Topic: topicCDR Partition: 7 Leader: 2 Replicas: 2,8,1 Isr: 8,2
> > > > > > Topic: topicCDR Partition: 8 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
> > > > > > Topic: topicCDR Partition: 9 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
> > > > > > Topic: topicCDR Partition: 10 Leader: 5 Replicas: 5,4,6 Isr:
> 4,5,6
> > > > > > Topic: topicCDR Partition: 11 Leader: 6 Replicas: 6,5,7 Isr:
> 5,6,7
> > > > > > Topic: topicCDR Partition: 12 Leader: 7 Replicas: 7,6,8 Isr:
> 6,7,8
> > > > > > Topic: topicCDR Partition: 13 Leader: 8 Replicas: 8,7,1 Isr: 7,8
> > > > > > Topic: topicCDR Partition: 14 Leader: 8 Replicas: 1,8,2 Isr: 8,2
> > > > > > Topic: topicCDR Partition: 15 Leader: 2 Replicas: 2,1,3 Isr: 3,2
> > > > > > Topic:topicDEBUG PartitionCount:16 ReplicationFactor:3 Configs:
> > > > > > retention.ms
> > > > > > =3600000
> > > > > > Topic: topicDEBUG Partition: 0 Leader: 4 Replicas: 4,3,5 Isr:
> 3,4,5
> > > > > > Topic: topicDEBUG Partition: 1 Leader: 5 Replicas: 5,4,6 Isr:
> 4,5,6
> > > > > > Topic: topicDEBUG Partition: 2 Leader: 6 Replicas: 6,5,7 Isr:
> 5,6,7
> > > > > > Topic: topicDEBUG Partition: 3 Leader: 7 Replicas: 7,6,8 Isr:
> 6,7,8
> > > > > > Topic: topicDEBUG Partition: 4 Leader: 8 Replicas: 8,7,1 Isr: 7,8
> > > > > > Topic: topicDEBUG Partition: 5 Leader: 8 Replicas: 1,8,2 Isr: 8,2
> > > > > > Topic: topicDEBUG Partition: 6 Leader: 2 Replicas: 2,1,3 Isr: 3,2
> > > > > > Topic: topicDEBUG Partition: 7 Leader: 3 Replicas: 3,2,4 Isr:
> 3,4,2
> > > > > > Topic: topicDEBUG Partition: 8 Leader: 4 Replicas: 4,5,6 Isr:
> 4,5,6
> > > > > > Topic: topicDEBUG Partition: 9 Leader: 5 Replicas: 5,6,7 Isr:
> 5,6,7
> > > > > > Topic: topicDEBUG Partition: 10 Leader: 6 Replicas: 6,7,8 Isr:
> > 6,7,8
> > > > > > Topic: topicDEBUG Partition: 11 Leader: 7 Replicas: 7,8,1 Isr:
> > 7,8,1
> > > > > > Topic: topicDEBUG Partition: 12 Leader: 8 Replicas: 8,1,2 Isr:
> 8,2
> > > > > > Topic: topicDEBUG Partition: 13 Leader: 3 Replicas: 1,2,3 Isr:
> 3,2
> > > > > > Topic: topicDEBUG Partition: 14 Leader: 2 Replicas: 2,3,4 Isr:
> > 3,4,2
> > > > > > Topic: topicDEBUG Partition: 15 Leader: 3 Replicas: 3,4,5 Isr:
> > 3,4,5
> > > > > > Topic:topicTRACE PartitionCount:16 ReplicationFactor:3 Configs:
> > > > > > retention.ms
> > > > > > =3600000
> > > > > > Topic: topicTRACE Partition: 0 Leader: 5 Replicas: 5,8,1 Isr:
> 5,8,1
> > > > > > Topic: topicTRACE Partition: 1 Leader: 6 Replicas: 6,1,2 Isr:
> 6,1,2
> > > > > > Topic: topicTRACE Partition: 2 Leader: 7 Replicas: 7,2,3 Isr:
> 3,7,2
> > > > > > Topic: topicTRACE Partition: 3 Leader: 8 Replicas: 8,3,4 Isr:
> 3,4,8
> > > > > > Topic: topicTRACE Partition: 4 Leader: 1 Replicas: 1,4,5 Isr:
> 1,5,4
> > > > > > Topic: topicTRACE Partition: 5 Leader: 2 Replicas: 2,5,6 Isr:
> 5,6,2
> > > > > > Topic: topicTRACE Partition: 6 Leader: 3 Replicas: 3,6,7 Isr:
> 3,6,7
> > > > > > Topic: topicTRACE Partition: 7 Leader: 4 Replicas: 4,7,8 Isr:
> 4,7,8
> > > > > > Topic: topicTRACE Partition: 8 Leader: 5 Replicas: 5,1,2 Isr:
> 5,1,2
> > > > > > Topic: topicTRACE Partition: 9 Leader: 6 Replicas: 6,2,3 Isr:
> 3,6,2
> > > > > > Topic: topicTRACE Partition: 10 Leader: 7 Replicas: 7,3,4 Isr:
> > 3,4,7
> > > > > > Topic: topicTRACE Partition: 11 Leader: 8 Replicas: 8,4,5 Isr:
> > 4,5,8
> > > > > >
> > > > > >
> > > > > > Problem is one of my topic's ISR is not updating and keep failing
> > to
> > > be
> > > > > > preferred replica. more detail, broker 1 for topicDEBUG's ISR is
> > not
> > > > > > updating.
> > > > > > And log of broker 1 is absolutely normal and has no error.
> > > > > >
> > > > > > This is expected situation? what I have to updating this?
> > > > > >
> > > > > >
> > > > > > Thanks in advance.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Sincerely*
> > > > > > *,**Bongyeon Kim*
> > > > > >
> > > > > > Java Developer & Engineer
> > > > > > Seoul, Korea
> > > > > > Mobile: +82-10-9369-1314
> > > > > > Email: bongyeonkim@gmail.com
> > > > > > Twitter: http://twitter.com/tigerby
> > > > > > Facebook: http://facebook.com/tigerby
> > > > > > Wiki: http://tigerby.com
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Sincerely*
> > > > *,**Bongyeon Kim*
> > > >
> > > > Java Developer & Engineer
> > > > Seoul, Korea
> > > > Mobile: +82-10-9369-1314
> > > > Email: bongyeonkim@gmail.com
> > > > Twitter: http://twitter.com/tigerby
> > > > Facebook: http://facebook.com/tigerby
> > > > Wiki: http://tigerby.com
> > > >
> > >
> >
> >
> >
> > --
> > *Sincerely*
> > *,**Bongyeon Kim*
> >
> > Java Developer & Engineer
> > Seoul, Korea
> > Mobile: +82-10-9369-1314
> > Email: bongyeonkim@gmail.com
> > Twitter: http://twitter.com/tigerby
> > Facebook: http://facebook.com/tigerby
> > Wiki: http://tigerby.com
> >
>
--
*Sincerely*
*,**Bongyeon Kim*
Java Developer & Engineer
Seoul, Korea
Mobile: +82-10-9369-1314
Email: bongyeonkim@gmail.com
Twitter: http://twitter.com/tigerby
Facebook: http://facebook.com/tigerby
Wiki: http://tigerby.com
Re: ISR not updating
Posted by Jun Rao <ju...@gmail.com>.
Auto leader balancing has some know issues when using together with
controlled shutdown. So, we don't recommend it to be turned on in 0.8.1.1
Thanks,
Jun
On Wed, Jun 18, 2014 at 1:41 AM, Bongyeon Kim <bo...@gmail.com>
wrote:
> Yes.
> it contain my server.properties file.
>
>
> auto.leader.rebalance.enable=true
>
>
> On Wed, Jun 18, 2014 at 12:44 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Did you have auto leader balancing enabled?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jun 17, 2014 at 5:06 PM, Bongyeon Kim <bo...@gmail.com>
> > wrote:
> >
> > > There is some error log about failing leader election like that.
> > >
> > >
> > > [2014-06-18 08:59:21,014] ERROR Controller 7 epoch 4 encountered error
> > > while electing leader for partition [topicDEBUG,5] due to: Preferred
> > > replica 1 for partition [topicDEBUG,5] is either not alive or not in
> the
> > > isr. Current leader and ISR:
> [{"leader":8,"leader_epoch":6,"isr":[8,2]}].
> > > (state.change.logger)
> > > [2014-06-18 08:59:21,014] ERROR Controller 7 epoch 4 initiated state
> > change
> > > for partition [topicDEBUG,5] from OnlinePartition to OnlinePartition
> > failed
> > > (state.change.logger)
> > > kafka.common.StateChangeFailedException: encountered error while
> electing
> > > leader for partition [topicDEBUG,5] due to: Preferred replica 1 for
> > > partition [topicDEBUG,5] is either not alive or not in the isr. Current
> > > leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}].
> > > at
> > >
> > >
> >
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:360)
> > > at
> > >
> > >
> >
> kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:187)
> > > at
> > >
> > >
> >
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:125)
> > > at
> > >
> > >
> >
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:124)
> > > at scala.collection.immutable.Set$Set1.foreach(Set.scala:86)
> > > at
> > >
> > >
> >
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:124)
> > > at
> > >
> > >
> >
> kafka.controller.KafkaController.onPreferredReplicaElection(KafkaController.scala:618)
> > > at
> > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply$mcV$sp(KafkaController.scala:1118)
> > > at
> > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply(KafkaController.scala:1112)
> > > at
> > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply(KafkaController.scala:1112)
> > > at kafka.utils.Utils$.inLock(Utils.scala:538)
> > > at
> > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1109)
> > > at
> > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1107)
> > > at
> > >
> >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
> > > at
> > >
> >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
> > > at scala.collection.Iterator$class.foreach(Iterator.scala:772)
> > > at
> > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
> > > at
> > >
> >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
> > > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
> > > at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
> > > at
> > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1107)
> > > at
> > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1086)
> > > at
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> > > at
> > >
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> > > at
> > >
> > >
> >
> kafka.controller.KafkaController.kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance(KafkaController.scala:1086)
> > > at
> > >
> > >
> >
> kafka.controller.KafkaController$$anonfun$onControllerFailover$1.apply$mcV$sp(KafkaController.scala:324)
> > > at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100)
> > > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > > at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > > at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:744)
> > > Caused by: kafka.common.StateChangeFailedException: Preferred replica 1
> > for
> > > partition [topicDEBUG,5] is either not alive or not in the isr. Current
> > > leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}]
> > > at
> > >
> > >
> >
> kafka.controller.PreferredReplicaPartitionLeaderSelector.selectLeader(PartitionLeaderSelector.scala:144)
> > > at
> > >
> > >
> >
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:336)
> > > ... 33 more
> > >
> > >
> > >
> > > Thanks.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Jun 17, 2014 at 12:42 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Any error in the controller and state-change log?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Jun 16, 2014 at 6:05 PM, Bongyeon Kim <
> bongyeon.kim@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi, team.
> > > > >
> > > > > Im using Kafka 0.8.1.1.
> > > > > I'm running 8 brokers on 4 machine. (2 brokers on 1 machine) and I
> > > have 3
> > > > > topics each have 16 partitions and 3 replicas.
> > > > >
> > > > > kafka-topics describe is
> > > > >
> > > > > Topic:topicCDR PartitionCount:16 ReplicationFactor:3 Configs:
> > > > retention.ms
> > > > > =3600000
> > > > > Topic: topicCDR Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,2
> > > > > Topic: topicCDR Partition: 1 Leader: 4 Replicas: 4,2,3 Isr: 3,4,2
> > > > > Topic: topicCDR Partition: 2 Leader: 5 Replicas: 5,3,4 Isr: 3,4,5
> > > > > Topic: topicCDR Partition: 3 Leader: 6 Replicas: 6,4,5 Isr: 4,5,6
> > > > > Topic: topicCDR Partition: 4 Leader: 7 Replicas: 7,5,6 Isr: 5,6,7
> > > > > Topic: topicCDR Partition: 5 Leader: 8 Replicas: 8,6,7 Isr: 6,7,8
> > > > > Topic: topicCDR Partition: 6 Leader: 1 Replicas: 1,7,8 Isr: 1,7,8
> > > > > Topic: topicCDR Partition: 7 Leader: 2 Replicas: 2,8,1 Isr: 8,2
> > > > > Topic: topicCDR Partition: 8 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
> > > > > Topic: topicCDR Partition: 9 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
> > > > > Topic: topicCDR Partition: 10 Leader: 5 Replicas: 5,4,6 Isr: 4,5,6
> > > > > Topic: topicCDR Partition: 11 Leader: 6 Replicas: 6,5,7 Isr: 5,6,7
> > > > > Topic: topicCDR Partition: 12 Leader: 7 Replicas: 7,6,8 Isr: 6,7,8
> > > > > Topic: topicCDR Partition: 13 Leader: 8 Replicas: 8,7,1 Isr: 7,8
> > > > > Topic: topicCDR Partition: 14 Leader: 8 Replicas: 1,8,2 Isr: 8,2
> > > > > Topic: topicCDR Partition: 15 Leader: 2 Replicas: 2,1,3 Isr: 3,2
> > > > > Topic:topicDEBUG PartitionCount:16 ReplicationFactor:3 Configs:
> > > > > retention.ms
> > > > > =3600000
> > > > > Topic: topicDEBUG Partition: 0 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
> > > > > Topic: topicDEBUG Partition: 1 Leader: 5 Replicas: 5,4,6 Isr: 4,5,6
> > > > > Topic: topicDEBUG Partition: 2 Leader: 6 Replicas: 6,5,7 Isr: 5,6,7
> > > > > Topic: topicDEBUG Partition: 3 Leader: 7 Replicas: 7,6,8 Isr: 6,7,8
> > > > > Topic: topicDEBUG Partition: 4 Leader: 8 Replicas: 8,7,1 Isr: 7,8
> > > > > Topic: topicDEBUG Partition: 5 Leader: 8 Replicas: 1,8,2 Isr: 8,2
> > > > > Topic: topicDEBUG Partition: 6 Leader: 2 Replicas: 2,1,3 Isr: 3,2
> > > > > Topic: topicDEBUG Partition: 7 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
> > > > > Topic: topicDEBUG Partition: 8 Leader: 4 Replicas: 4,5,6 Isr: 4,5,6
> > > > > Topic: topicDEBUG Partition: 9 Leader: 5 Replicas: 5,6,7 Isr: 5,6,7
> > > > > Topic: topicDEBUG Partition: 10 Leader: 6 Replicas: 6,7,8 Isr:
> 6,7,8
> > > > > Topic: topicDEBUG Partition: 11 Leader: 7 Replicas: 7,8,1 Isr:
> 7,8,1
> > > > > Topic: topicDEBUG Partition: 12 Leader: 8 Replicas: 8,1,2 Isr: 8,2
> > > > > Topic: topicDEBUG Partition: 13 Leader: 3 Replicas: 1,2,3 Isr: 3,2
> > > > > Topic: topicDEBUG Partition: 14 Leader: 2 Replicas: 2,3,4 Isr:
> 3,4,2
> > > > > Topic: topicDEBUG Partition: 15 Leader: 3 Replicas: 3,4,5 Isr:
> 3,4,5
> > > > > Topic:topicTRACE PartitionCount:16 ReplicationFactor:3 Configs:
> > > > > retention.ms
> > > > > =3600000
> > > > > Topic: topicTRACE Partition: 0 Leader: 5 Replicas: 5,8,1 Isr: 5,8,1
> > > > > Topic: topicTRACE Partition: 1 Leader: 6 Replicas: 6,1,2 Isr: 6,1,2
> > > > > Topic: topicTRACE Partition: 2 Leader: 7 Replicas: 7,2,3 Isr: 3,7,2
> > > > > Topic: topicTRACE Partition: 3 Leader: 8 Replicas: 8,3,4 Isr: 3,4,8
> > > > > Topic: topicTRACE Partition: 4 Leader: 1 Replicas: 1,4,5 Isr: 1,5,4
> > > > > Topic: topicTRACE Partition: 5 Leader: 2 Replicas: 2,5,6 Isr: 5,6,2
> > > > > Topic: topicTRACE Partition: 6 Leader: 3 Replicas: 3,6,7 Isr: 3,6,7
> > > > > Topic: topicTRACE Partition: 7 Leader: 4 Replicas: 4,7,8 Isr: 4,7,8
> > > > > Topic: topicTRACE Partition: 8 Leader: 5 Replicas: 5,1,2 Isr: 5,1,2
> > > > > Topic: topicTRACE Partition: 9 Leader: 6 Replicas: 6,2,3 Isr: 3,6,2
> > > > > Topic: topicTRACE Partition: 10 Leader: 7 Replicas: 7,3,4 Isr:
> 3,4,7
> > > > > Topic: topicTRACE Partition: 11 Leader: 8 Replicas: 8,4,5 Isr:
> 4,5,8
> > > > >
> > > > >
> > > > > Problem is one of my topic's ISR is not updating and keep failing
> to
> > be
> > > > > preferred replica. more detail, broker 1 for topicDEBUG's ISR is
> not
> > > > > updating.
> > > > > And log of broker 1 is absolutely normal and has no error.
> > > > >
> > > > > This is expected situation? what I have to updating this?
> > > > >
> > > > >
> > > > > Thanks in advance.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Sincerely*
> > > > > *,**Bongyeon Kim*
> > > > >
> > > > > Java Developer & Engineer
> > > > > Seoul, Korea
> > > > > Mobile: +82-10-9369-1314
> > > > > Email: bongyeonkim@gmail.com
> > > > > Twitter: http://twitter.com/tigerby
> > > > > Facebook: http://facebook.com/tigerby
> > > > > Wiki: http://tigerby.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Sincerely*
> > > *,**Bongyeon Kim*
> > >
> > > Java Developer & Engineer
> > > Seoul, Korea
> > > Mobile: +82-10-9369-1314
> > > Email: bongyeonkim@gmail.com
> > > Twitter: http://twitter.com/tigerby
> > > Facebook: http://facebook.com/tigerby
> > > Wiki: http://tigerby.com
> > >
> >
>
>
>
> --
> *Sincerely*
> *,**Bongyeon Kim*
>
> Java Developer & Engineer
> Seoul, Korea
> Mobile: +82-10-9369-1314
> Email: bongyeonkim@gmail.com
> Twitter: http://twitter.com/tigerby
> Facebook: http://facebook.com/tigerby
> Wiki: http://tigerby.com
>
Re: ISR not updating
Posted by Bongyeon Kim <bo...@gmail.com>.
Yes.
it contain my server.properties file.
auto.leader.rebalance.enable=true
On Wed, Jun 18, 2014 at 12:44 PM, Jun Rao <ju...@gmail.com> wrote:
> Did you have auto leader balancing enabled?
>
> Thanks,
>
> Jun
>
>
> On Tue, Jun 17, 2014 at 5:06 PM, Bongyeon Kim <bo...@gmail.com>
> wrote:
>
> > There is some error log about failing leader election like that.
> >
> >
> > [2014-06-18 08:59:21,014] ERROR Controller 7 epoch 4 encountered error
> > while electing leader for partition [topicDEBUG,5] due to: Preferred
> > replica 1 for partition [topicDEBUG,5] is either not alive or not in the
> > isr. Current leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}].
> > (state.change.logger)
> > [2014-06-18 08:59:21,014] ERROR Controller 7 epoch 4 initiated state
> change
> > for partition [topicDEBUG,5] from OnlinePartition to OnlinePartition
> failed
> > (state.change.logger)
> > kafka.common.StateChangeFailedException: encountered error while electing
> > leader for partition [topicDEBUG,5] due to: Preferred replica 1 for
> > partition [topicDEBUG,5] is either not alive or not in the isr. Current
> > leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}].
> > at
> >
> >
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:360)
> > at
> >
> >
> kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:187)
> > at
> >
> >
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:125)
> > at
> >
> >
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:124)
> > at scala.collection.immutable.Set$Set1.foreach(Set.scala:86)
> > at
> >
> >
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:124)
> > at
> >
> >
> kafka.controller.KafkaController.onPreferredReplicaElection(KafkaController.scala:618)
> > at
> >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply$mcV$sp(KafkaController.scala:1118)
> > at
> >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply(KafkaController.scala:1112)
> > at
> >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply(KafkaController.scala:1112)
> > at kafka.utils.Utils$.inLock(Utils.scala:538)
> > at
> >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1109)
> > at
> >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1107)
> > at
> >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
> > at
> >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
> > at scala.collection.Iterator$class.foreach(Iterator.scala:772)
> > at
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
> > at
> >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
> > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
> > at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
> > at
> >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1107)
> > at
> >
> >
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1086)
> > at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> > at
> > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> > at
> >
> >
> kafka.controller.KafkaController.kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance(KafkaController.scala:1086)
> > at
> >
> >
> kafka.controller.KafkaController$$anonfun$onControllerFailover$1.apply$mcV$sp(KafkaController.scala:324)
> > at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:744)
> > Caused by: kafka.common.StateChangeFailedException: Preferred replica 1
> for
> > partition [topicDEBUG,5] is either not alive or not in the isr. Current
> > leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}]
> > at
> >
> >
> kafka.controller.PreferredReplicaPartitionLeaderSelector.selectLeader(PartitionLeaderSelector.scala:144)
> > at
> >
> >
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:336)
> > ... 33 more
> >
> >
> >
> > Thanks.
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Jun 17, 2014 at 12:42 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Any error in the controller and state-change log?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Jun 16, 2014 at 6:05 PM, Bongyeon Kim <bo...@gmail.com>
> > > wrote:
> > >
> > > > Hi, team.
> > > >
> > > > Im using Kafka 0.8.1.1.
> > > > I'm running 8 brokers on 4 machine. (2 brokers on 1 machine) and I
> > have 3
> > > > topics each have 16 partitions and 3 replicas.
> > > >
> > > > kafka-topics describe is
> > > >
> > > > Topic:topicCDR PartitionCount:16 ReplicationFactor:3 Configs:
> > > retention.ms
> > > > =3600000
> > > > Topic: topicCDR Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,2
> > > > Topic: topicCDR Partition: 1 Leader: 4 Replicas: 4,2,3 Isr: 3,4,2
> > > > Topic: topicCDR Partition: 2 Leader: 5 Replicas: 5,3,4 Isr: 3,4,5
> > > > Topic: topicCDR Partition: 3 Leader: 6 Replicas: 6,4,5 Isr: 4,5,6
> > > > Topic: topicCDR Partition: 4 Leader: 7 Replicas: 7,5,6 Isr: 5,6,7
> > > > Topic: topicCDR Partition: 5 Leader: 8 Replicas: 8,6,7 Isr: 6,7,8
> > > > Topic: topicCDR Partition: 6 Leader: 1 Replicas: 1,7,8 Isr: 1,7,8
> > > > Topic: topicCDR Partition: 7 Leader: 2 Replicas: 2,8,1 Isr: 8,2
> > > > Topic: topicCDR Partition: 8 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
> > > > Topic: topicCDR Partition: 9 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
> > > > Topic: topicCDR Partition: 10 Leader: 5 Replicas: 5,4,6 Isr: 4,5,6
> > > > Topic: topicCDR Partition: 11 Leader: 6 Replicas: 6,5,7 Isr: 5,6,7
> > > > Topic: topicCDR Partition: 12 Leader: 7 Replicas: 7,6,8 Isr: 6,7,8
> > > > Topic: topicCDR Partition: 13 Leader: 8 Replicas: 8,7,1 Isr: 7,8
> > > > Topic: topicCDR Partition: 14 Leader: 8 Replicas: 1,8,2 Isr: 8,2
> > > > Topic: topicCDR Partition: 15 Leader: 2 Replicas: 2,1,3 Isr: 3,2
> > > > Topic:topicDEBUG PartitionCount:16 ReplicationFactor:3 Configs:
> > > > retention.ms
> > > > =3600000
> > > > Topic: topicDEBUG Partition: 0 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
> > > > Topic: topicDEBUG Partition: 1 Leader: 5 Replicas: 5,4,6 Isr: 4,5,6
> > > > Topic: topicDEBUG Partition: 2 Leader: 6 Replicas: 6,5,7 Isr: 5,6,7
> > > > Topic: topicDEBUG Partition: 3 Leader: 7 Replicas: 7,6,8 Isr: 6,7,8
> > > > Topic: topicDEBUG Partition: 4 Leader: 8 Replicas: 8,7,1 Isr: 7,8
> > > > Topic: topicDEBUG Partition: 5 Leader: 8 Replicas: 1,8,2 Isr: 8,2
> > > > Topic: topicDEBUG Partition: 6 Leader: 2 Replicas: 2,1,3 Isr: 3,2
> > > > Topic: topicDEBUG Partition: 7 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
> > > > Topic: topicDEBUG Partition: 8 Leader: 4 Replicas: 4,5,6 Isr: 4,5,6
> > > > Topic: topicDEBUG Partition: 9 Leader: 5 Replicas: 5,6,7 Isr: 5,6,7
> > > > Topic: topicDEBUG Partition: 10 Leader: 6 Replicas: 6,7,8 Isr: 6,7,8
> > > > Topic: topicDEBUG Partition: 11 Leader: 7 Replicas: 7,8,1 Isr: 7,8,1
> > > > Topic: topicDEBUG Partition: 12 Leader: 8 Replicas: 8,1,2 Isr: 8,2
> > > > Topic: topicDEBUG Partition: 13 Leader: 3 Replicas: 1,2,3 Isr: 3,2
> > > > Topic: topicDEBUG Partition: 14 Leader: 2 Replicas: 2,3,4 Isr: 3,4,2
> > > > Topic: topicDEBUG Partition: 15 Leader: 3 Replicas: 3,4,5 Isr: 3,4,5
> > > > Topic:topicTRACE PartitionCount:16 ReplicationFactor:3 Configs:
> > > > retention.ms
> > > > =3600000
> > > > Topic: topicTRACE Partition: 0 Leader: 5 Replicas: 5,8,1 Isr: 5,8,1
> > > > Topic: topicTRACE Partition: 1 Leader: 6 Replicas: 6,1,2 Isr: 6,1,2
> > > > Topic: topicTRACE Partition: 2 Leader: 7 Replicas: 7,2,3 Isr: 3,7,2
> > > > Topic: topicTRACE Partition: 3 Leader: 8 Replicas: 8,3,4 Isr: 3,4,8
> > > > Topic: topicTRACE Partition: 4 Leader: 1 Replicas: 1,4,5 Isr: 1,5,4
> > > > Topic: topicTRACE Partition: 5 Leader: 2 Replicas: 2,5,6 Isr: 5,6,2
> > > > Topic: topicTRACE Partition: 6 Leader: 3 Replicas: 3,6,7 Isr: 3,6,7
> > > > Topic: topicTRACE Partition: 7 Leader: 4 Replicas: 4,7,8 Isr: 4,7,8
> > > > Topic: topicTRACE Partition: 8 Leader: 5 Replicas: 5,1,2 Isr: 5,1,2
> > > > Topic: topicTRACE Partition: 9 Leader: 6 Replicas: 6,2,3 Isr: 3,6,2
> > > > Topic: topicTRACE Partition: 10 Leader: 7 Replicas: 7,3,4 Isr: 3,4,7
> > > > Topic: topicTRACE Partition: 11 Leader: 8 Replicas: 8,4,5 Isr: 4,5,8
> > > >
> > > >
> > > > Problem is one of my topic's ISR is not updating and keep failing to
> be
> > > > preferred replica. more detail, broker 1 for topicDEBUG's ISR is not
> > > > updating.
> > > > And log of broker 1 is absolutely normal and has no error.
> > > >
> > > > This is expected situation? what I have to updating this?
> > > >
> > > >
> > > > Thanks in advance.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Sincerely*
> > > > *,**Bongyeon Kim*
> > > >
> > > > Java Developer & Engineer
> > > > Seoul, Korea
> > > > Mobile: +82-10-9369-1314
> > > > Email: bongyeonkim@gmail.com
> > > > Twitter: http://twitter.com/tigerby
> > > > Facebook: http://facebook.com/tigerby
> > > > Wiki: http://tigerby.com
> > > >
> > >
> >
> >
> >
> > --
> > *Sincerely*
> > *,**Bongyeon Kim*
> >
> > Java Developer & Engineer
> > Seoul, Korea
> > Mobile: +82-10-9369-1314
> > Email: bongyeonkim@gmail.com
> > Twitter: http://twitter.com/tigerby
> > Facebook: http://facebook.com/tigerby
> > Wiki: http://tigerby.com
> >
>
--
*Sincerely*
*,**Bongyeon Kim*
Java Developer & Engineer
Seoul, Korea
Mobile: +82-10-9369-1314
Email: bongyeonkim@gmail.com
Twitter: http://twitter.com/tigerby
Facebook: http://facebook.com/tigerby
Wiki: http://tigerby.com
Re: ISR not updating
Posted by Jun Rao <ju...@gmail.com>.
Did you have auto leader balancing enabled?
Thanks,
Jun
On Tue, Jun 17, 2014 at 5:06 PM, Bongyeon Kim <bo...@gmail.com>
wrote:
> There is some error log about failing leader election like that.
>
>
> [2014-06-18 08:59:21,014] ERROR Controller 7 epoch 4 encountered error
> while electing leader for partition [topicDEBUG,5] due to: Preferred
> replica 1 for partition [topicDEBUG,5] is either not alive or not in the
> isr. Current leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}].
> (state.change.logger)
> [2014-06-18 08:59:21,014] ERROR Controller 7 epoch 4 initiated state change
> for partition [topicDEBUG,5] from OnlinePartition to OnlinePartition failed
> (state.change.logger)
> kafka.common.StateChangeFailedException: encountered error while electing
> leader for partition [topicDEBUG,5] due to: Preferred replica 1 for
> partition [topicDEBUG,5] is either not alive or not in the isr. Current
> leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}].
> at
>
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:360)
> at
>
> kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:187)
> at
>
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:125)
> at
>
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:124)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:86)
> at
>
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:124)
> at
>
> kafka.controller.KafkaController.onPreferredReplicaElection(KafkaController.scala:618)
> at
>
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply$mcV$sp(KafkaController.scala:1118)
> at
>
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply(KafkaController.scala:1112)
> at
>
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply(KafkaController.scala:1112)
> at kafka.utils.Utils$.inLock(Utils.scala:538)
> at
>
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1109)
> at
>
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1107)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
> at scala.collection.Iterator$class.foreach(Iterator.scala:772)
> at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
> at
>
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1107)
> at
>
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1086)
> at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> at
>
> kafka.controller.KafkaController.kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance(KafkaController.scala:1086)
> at
>
> kafka.controller.KafkaController$$anonfun$onControllerFailover$1.apply$mcV$sp(KafkaController.scala:324)
> at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: kafka.common.StateChangeFailedException: Preferred replica 1 for
> partition [topicDEBUG,5] is either not alive or not in the isr. Current
> leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}]
> at
>
> kafka.controller.PreferredReplicaPartitionLeaderSelector.selectLeader(PartitionLeaderSelector.scala:144)
> at
>
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:336)
> ... 33 more
>
>
>
> Thanks.
>
>
>
>
>
>
>
>
> On Tue, Jun 17, 2014 at 12:42 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Any error in the controller and state-change log?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Jun 16, 2014 at 6:05 PM, Bongyeon Kim <bo...@gmail.com>
> > wrote:
> >
> > > Hi, team.
> > >
> > > Im using Kafka 0.8.1.1.
> > > I'm running 8 brokers on 4 machine. (2 brokers on 1 machine) and I
> have 3
> > > topics each have 16 partitions and 3 replicas.
> > >
> > > kafka-topics describe is
> > >
> > > Topic:topicCDR PartitionCount:16 ReplicationFactor:3 Configs:
> > retention.ms
> > > =3600000
> > > Topic: topicCDR Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,2
> > > Topic: topicCDR Partition: 1 Leader: 4 Replicas: 4,2,3 Isr: 3,4,2
> > > Topic: topicCDR Partition: 2 Leader: 5 Replicas: 5,3,4 Isr: 3,4,5
> > > Topic: topicCDR Partition: 3 Leader: 6 Replicas: 6,4,5 Isr: 4,5,6
> > > Topic: topicCDR Partition: 4 Leader: 7 Replicas: 7,5,6 Isr: 5,6,7
> > > Topic: topicCDR Partition: 5 Leader: 8 Replicas: 8,6,7 Isr: 6,7,8
> > > Topic: topicCDR Partition: 6 Leader: 1 Replicas: 1,7,8 Isr: 1,7,8
> > > Topic: topicCDR Partition: 7 Leader: 2 Replicas: 2,8,1 Isr: 8,2
> > > Topic: topicCDR Partition: 8 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
> > > Topic: topicCDR Partition: 9 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
> > > Topic: topicCDR Partition: 10 Leader: 5 Replicas: 5,4,6 Isr: 4,5,6
> > > Topic: topicCDR Partition: 11 Leader: 6 Replicas: 6,5,7 Isr: 5,6,7
> > > Topic: topicCDR Partition: 12 Leader: 7 Replicas: 7,6,8 Isr: 6,7,8
> > > Topic: topicCDR Partition: 13 Leader: 8 Replicas: 8,7,1 Isr: 7,8
> > > Topic: topicCDR Partition: 14 Leader: 8 Replicas: 1,8,2 Isr: 8,2
> > > Topic: topicCDR Partition: 15 Leader: 2 Replicas: 2,1,3 Isr: 3,2
> > > Topic:topicDEBUG PartitionCount:16 ReplicationFactor:3 Configs:
> > > retention.ms
> > > =3600000
> > > Topic: topicDEBUG Partition: 0 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
> > > Topic: topicDEBUG Partition: 1 Leader: 5 Replicas: 5,4,6 Isr: 4,5,6
> > > Topic: topicDEBUG Partition: 2 Leader: 6 Replicas: 6,5,7 Isr: 5,6,7
> > > Topic: topicDEBUG Partition: 3 Leader: 7 Replicas: 7,6,8 Isr: 6,7,8
> > > Topic: topicDEBUG Partition: 4 Leader: 8 Replicas: 8,7,1 Isr: 7,8
> > > Topic: topicDEBUG Partition: 5 Leader: 8 Replicas: 1,8,2 Isr: 8,2
> > > Topic: topicDEBUG Partition: 6 Leader: 2 Replicas: 2,1,3 Isr: 3,2
> > > Topic: topicDEBUG Partition: 7 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
> > > Topic: topicDEBUG Partition: 8 Leader: 4 Replicas: 4,5,6 Isr: 4,5,6
> > > Topic: topicDEBUG Partition: 9 Leader: 5 Replicas: 5,6,7 Isr: 5,6,7
> > > Topic: topicDEBUG Partition: 10 Leader: 6 Replicas: 6,7,8 Isr: 6,7,8
> > > Topic: topicDEBUG Partition: 11 Leader: 7 Replicas: 7,8,1 Isr: 7,8,1
> > > Topic: topicDEBUG Partition: 12 Leader: 8 Replicas: 8,1,2 Isr: 8,2
> > > Topic: topicDEBUG Partition: 13 Leader: 3 Replicas: 1,2,3 Isr: 3,2
> > > Topic: topicDEBUG Partition: 14 Leader: 2 Replicas: 2,3,4 Isr: 3,4,2
> > > Topic: topicDEBUG Partition: 15 Leader: 3 Replicas: 3,4,5 Isr: 3,4,5
> > > Topic:topicTRACE PartitionCount:16 ReplicationFactor:3 Configs:
> > > retention.ms
> > > =3600000
> > > Topic: topicTRACE Partition: 0 Leader: 5 Replicas: 5,8,1 Isr: 5,8,1
> > > Topic: topicTRACE Partition: 1 Leader: 6 Replicas: 6,1,2 Isr: 6,1,2
> > > Topic: topicTRACE Partition: 2 Leader: 7 Replicas: 7,2,3 Isr: 3,7,2
> > > Topic: topicTRACE Partition: 3 Leader: 8 Replicas: 8,3,4 Isr: 3,4,8
> > > Topic: topicTRACE Partition: 4 Leader: 1 Replicas: 1,4,5 Isr: 1,5,4
> > > Topic: topicTRACE Partition: 5 Leader: 2 Replicas: 2,5,6 Isr: 5,6,2
> > > Topic: topicTRACE Partition: 6 Leader: 3 Replicas: 3,6,7 Isr: 3,6,7
> > > Topic: topicTRACE Partition: 7 Leader: 4 Replicas: 4,7,8 Isr: 4,7,8
> > > Topic: topicTRACE Partition: 8 Leader: 5 Replicas: 5,1,2 Isr: 5,1,2
> > > Topic: topicTRACE Partition: 9 Leader: 6 Replicas: 6,2,3 Isr: 3,6,2
> > > Topic: topicTRACE Partition: 10 Leader: 7 Replicas: 7,3,4 Isr: 3,4,7
> > > Topic: topicTRACE Partition: 11 Leader: 8 Replicas: 8,4,5 Isr: 4,5,8
> > >
> > >
> > > Problem is one of my topic's ISR is not updating and keep failing to be
> > > preferred replica. more detail, broker 1 for topicDEBUG's ISR is not
> > > updating.
> > > And log of broker 1 is absolutely normal and has no error.
> > >
> > > This is expected situation? what I have to updating this?
> > >
> > >
> > > Thanks in advance.
> > >
> > >
> > >
> > >
> > >
> > > --
> > > *Sincerely*
> > > *,**Bongyeon Kim*
> > >
> > > Java Developer & Engineer
> > > Seoul, Korea
> > > Mobile: +82-10-9369-1314
> > > Email: bongyeonkim@gmail.com
> > > Twitter: http://twitter.com/tigerby
> > > Facebook: http://facebook.com/tigerby
> > > Wiki: http://tigerby.com
> > >
> >
>
>
>
> --
> *Sincerely*
> *,**Bongyeon Kim*
>
> Java Developer & Engineer
> Seoul, Korea
> Mobile: +82-10-9369-1314
> Email: bongyeonkim@gmail.com
> Twitter: http://twitter.com/tigerby
> Facebook: http://facebook.com/tigerby
> Wiki: http://tigerby.com
>
Re: ISR not updating
Posted by Bongyeon Kim <bo...@gmail.com>.
There is some error log about failing leader election like that.
[2014-06-18 08:59:21,014] ERROR Controller 7 epoch 4 encountered error
while electing leader for partition [topicDEBUG,5] due to: Preferred
replica 1 for partition [topicDEBUG,5] is either not alive or not in the
isr. Current leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}].
(state.change.logger)
[2014-06-18 08:59:21,014] ERROR Controller 7 epoch 4 initiated state change
for partition [topicDEBUG,5] from OnlinePartition to OnlinePartition failed
(state.change.logger)
kafka.common.StateChangeFailedException: encountered error while electing
leader for partition [topicDEBUG,5] due to: Preferred replica 1 for
partition [topicDEBUG,5] is either not alive or not in the isr. Current
leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}].
at
kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:360)
at
kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:187)
at
kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:125)
at
kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:124)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:86)
at
kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:124)
at
kafka.controller.KafkaController.onPreferredReplicaElection(KafkaController.scala:618)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply$mcV$sp(KafkaController.scala:1118)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply(KafkaController.scala:1112)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17$$anonfun$apply$5.apply(KafkaController.scala:1112)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1109)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1107)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1107)
at
kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1086)
at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
at
kafka.controller.KafkaController.kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance(KafkaController.scala:1086)
at
kafka.controller.KafkaController$$anonfun$onControllerFailover$1.apply$mcV$sp(KafkaController.scala:324)
at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: kafka.common.StateChangeFailedException: Preferred replica 1 for
partition [topicDEBUG,5] is either not alive or not in the isr. Current
leader and ISR: [{"leader":8,"leader_epoch":6,"isr":[8,2]}]
at
kafka.controller.PreferredReplicaPartitionLeaderSelector.selectLeader(PartitionLeaderSelector.scala:144)
at
kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:336)
... 33 more
Thanks.
On Tue, Jun 17, 2014 at 12:42 PM, Jun Rao <ju...@gmail.com> wrote:
> Any error in the controller and state-change log?
>
> Thanks,
>
> Jun
>
>
> On Mon, Jun 16, 2014 at 6:05 PM, Bongyeon Kim <bo...@gmail.com>
> wrote:
>
> > Hi, team.
> >
> > Im using Kafka 0.8.1.1.
> > I'm running 8 brokers on 4 machine. (2 brokers on 1 machine) and I have 3
> > topics each have 16 partitions and 3 replicas.
> >
> > kafka-topics describe is
> >
> > Topic:topicCDR PartitionCount:16 ReplicationFactor:3 Configs:
> retention.ms
> > =3600000
> > Topic: topicCDR Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,2
> > Topic: topicCDR Partition: 1 Leader: 4 Replicas: 4,2,3 Isr: 3,4,2
> > Topic: topicCDR Partition: 2 Leader: 5 Replicas: 5,3,4 Isr: 3,4,5
> > Topic: topicCDR Partition: 3 Leader: 6 Replicas: 6,4,5 Isr: 4,5,6
> > Topic: topicCDR Partition: 4 Leader: 7 Replicas: 7,5,6 Isr: 5,6,7
> > Topic: topicCDR Partition: 5 Leader: 8 Replicas: 8,6,7 Isr: 6,7,8
> > Topic: topicCDR Partition: 6 Leader: 1 Replicas: 1,7,8 Isr: 1,7,8
> > Topic: topicCDR Partition: 7 Leader: 2 Replicas: 2,8,1 Isr: 8,2
> > Topic: topicCDR Partition: 8 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
> > Topic: topicCDR Partition: 9 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
> > Topic: topicCDR Partition: 10 Leader: 5 Replicas: 5,4,6 Isr: 4,5,6
> > Topic: topicCDR Partition: 11 Leader: 6 Replicas: 6,5,7 Isr: 5,6,7
> > Topic: topicCDR Partition: 12 Leader: 7 Replicas: 7,6,8 Isr: 6,7,8
> > Topic: topicCDR Partition: 13 Leader: 8 Replicas: 8,7,1 Isr: 7,8
> > Topic: topicCDR Partition: 14 Leader: 8 Replicas: 1,8,2 Isr: 8,2
> > Topic: topicCDR Partition: 15 Leader: 2 Replicas: 2,1,3 Isr: 3,2
> > Topic:topicDEBUG PartitionCount:16 ReplicationFactor:3 Configs:
> > retention.ms
> > =3600000
> > Topic: topicDEBUG Partition: 0 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
> > Topic: topicDEBUG Partition: 1 Leader: 5 Replicas: 5,4,6 Isr: 4,5,6
> > Topic: topicDEBUG Partition: 2 Leader: 6 Replicas: 6,5,7 Isr: 5,6,7
> > Topic: topicDEBUG Partition: 3 Leader: 7 Replicas: 7,6,8 Isr: 6,7,8
> > Topic: topicDEBUG Partition: 4 Leader: 8 Replicas: 8,7,1 Isr: 7,8
> > Topic: topicDEBUG Partition: 5 Leader: 8 Replicas: 1,8,2 Isr: 8,2
> > Topic: topicDEBUG Partition: 6 Leader: 2 Replicas: 2,1,3 Isr: 3,2
> > Topic: topicDEBUG Partition: 7 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
> > Topic: topicDEBUG Partition: 8 Leader: 4 Replicas: 4,5,6 Isr: 4,5,6
> > Topic: topicDEBUG Partition: 9 Leader: 5 Replicas: 5,6,7 Isr: 5,6,7
> > Topic: topicDEBUG Partition: 10 Leader: 6 Replicas: 6,7,8 Isr: 6,7,8
> > Topic: topicDEBUG Partition: 11 Leader: 7 Replicas: 7,8,1 Isr: 7,8,1
> > Topic: topicDEBUG Partition: 12 Leader: 8 Replicas: 8,1,2 Isr: 8,2
> > Topic: topicDEBUG Partition: 13 Leader: 3 Replicas: 1,2,3 Isr: 3,2
> > Topic: topicDEBUG Partition: 14 Leader: 2 Replicas: 2,3,4 Isr: 3,4,2
> > Topic: topicDEBUG Partition: 15 Leader: 3 Replicas: 3,4,5 Isr: 3,4,5
> > Topic:topicTRACE PartitionCount:16 ReplicationFactor:3 Configs:
> > retention.ms
> > =3600000
> > Topic: topicTRACE Partition: 0 Leader: 5 Replicas: 5,8,1 Isr: 5,8,1
> > Topic: topicTRACE Partition: 1 Leader: 6 Replicas: 6,1,2 Isr: 6,1,2
> > Topic: topicTRACE Partition: 2 Leader: 7 Replicas: 7,2,3 Isr: 3,7,2
> > Topic: topicTRACE Partition: 3 Leader: 8 Replicas: 8,3,4 Isr: 3,4,8
> > Topic: topicTRACE Partition: 4 Leader: 1 Replicas: 1,4,5 Isr: 1,5,4
> > Topic: topicTRACE Partition: 5 Leader: 2 Replicas: 2,5,6 Isr: 5,6,2
> > Topic: topicTRACE Partition: 6 Leader: 3 Replicas: 3,6,7 Isr: 3,6,7
> > Topic: topicTRACE Partition: 7 Leader: 4 Replicas: 4,7,8 Isr: 4,7,8
> > Topic: topicTRACE Partition: 8 Leader: 5 Replicas: 5,1,2 Isr: 5,1,2
> > Topic: topicTRACE Partition: 9 Leader: 6 Replicas: 6,2,3 Isr: 3,6,2
> > Topic: topicTRACE Partition: 10 Leader: 7 Replicas: 7,3,4 Isr: 3,4,7
> > Topic: topicTRACE Partition: 11 Leader: 8 Replicas: 8,4,5 Isr: 4,5,8
> >
> >
> > Problem is one of my topic's ISR is not updating and keep failing to be
> > preferred replica. more detail, broker 1 for topicDEBUG's ISR is not
> > updating.
> > And log of broker 1 is absolutely normal and has no error.
> >
> > This is expected situation? what I have to updating this?
> >
> >
> > Thanks in advance.
> >
> >
> >
> >
> >
> > --
> > *Sincerely*
> > *,**Bongyeon Kim*
> >
> > Java Developer & Engineer
> > Seoul, Korea
> > Mobile: +82-10-9369-1314
> > Email: bongyeonkim@gmail.com
> > Twitter: http://twitter.com/tigerby
> > Facebook: http://facebook.com/tigerby
> > Wiki: http://tigerby.com
> >
>
--
*Sincerely*
*,**Bongyeon Kim*
Java Developer & Engineer
Seoul, Korea
Mobile: +82-10-9369-1314
Email: bongyeonkim@gmail.com
Twitter: http://twitter.com/tigerby
Facebook: http://facebook.com/tigerby
Wiki: http://tigerby.com
Re: ISR not updating
Posted by Jun Rao <ju...@gmail.com>.
Any error in the controller and state-change log?
Thanks,
Jun
On Mon, Jun 16, 2014 at 6:05 PM, Bongyeon Kim <bo...@gmail.com>
wrote:
> Hi, team.
>
> Im using Kafka 0.8.1.1.
> I'm running 8 brokers on 4 machine. (2 brokers on 1 machine) and I have 3
> topics each have 16 partitions and 3 replicas.
>
> kafka-topics describe is
>
> Topic:topicCDR PartitionCount:16 ReplicationFactor:3 Configs:retention.ms
> =3600000
> Topic: topicCDR Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,2
> Topic: topicCDR Partition: 1 Leader: 4 Replicas: 4,2,3 Isr: 3,4,2
> Topic: topicCDR Partition: 2 Leader: 5 Replicas: 5,3,4 Isr: 3,4,5
> Topic: topicCDR Partition: 3 Leader: 6 Replicas: 6,4,5 Isr: 4,5,6
> Topic: topicCDR Partition: 4 Leader: 7 Replicas: 7,5,6 Isr: 5,6,7
> Topic: topicCDR Partition: 5 Leader: 8 Replicas: 8,6,7 Isr: 6,7,8
> Topic: topicCDR Partition: 6 Leader: 1 Replicas: 1,7,8 Isr: 1,7,8
> Topic: topicCDR Partition: 7 Leader: 2 Replicas: 2,8,1 Isr: 8,2
> Topic: topicCDR Partition: 8 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
> Topic: topicCDR Partition: 9 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
> Topic: topicCDR Partition: 10 Leader: 5 Replicas: 5,4,6 Isr: 4,5,6
> Topic: topicCDR Partition: 11 Leader: 6 Replicas: 6,5,7 Isr: 5,6,7
> Topic: topicCDR Partition: 12 Leader: 7 Replicas: 7,6,8 Isr: 6,7,8
> Topic: topicCDR Partition: 13 Leader: 8 Replicas: 8,7,1 Isr: 7,8
> Topic: topicCDR Partition: 14 Leader: 8 Replicas: 1,8,2 Isr: 8,2
> Topic: topicCDR Partition: 15 Leader: 2 Replicas: 2,1,3 Isr: 3,2
> Topic:topicDEBUG PartitionCount:16 ReplicationFactor:3 Configs:
> retention.ms
> =3600000
> Topic: topicDEBUG Partition: 0 Leader: 4 Replicas: 4,3,5 Isr: 3,4,5
> Topic: topicDEBUG Partition: 1 Leader: 5 Replicas: 5,4,6 Isr: 4,5,6
> Topic: topicDEBUG Partition: 2 Leader: 6 Replicas: 6,5,7 Isr: 5,6,7
> Topic: topicDEBUG Partition: 3 Leader: 7 Replicas: 7,6,8 Isr: 6,7,8
> Topic: topicDEBUG Partition: 4 Leader: 8 Replicas: 8,7,1 Isr: 7,8
> Topic: topicDEBUG Partition: 5 Leader: 8 Replicas: 1,8,2 Isr: 8,2
> Topic: topicDEBUG Partition: 6 Leader: 2 Replicas: 2,1,3 Isr: 3,2
> Topic: topicDEBUG Partition: 7 Leader: 3 Replicas: 3,2,4 Isr: 3,4,2
> Topic: topicDEBUG Partition: 8 Leader: 4 Replicas: 4,5,6 Isr: 4,5,6
> Topic: topicDEBUG Partition: 9 Leader: 5 Replicas: 5,6,7 Isr: 5,6,7
> Topic: topicDEBUG Partition: 10 Leader: 6 Replicas: 6,7,8 Isr: 6,7,8
> Topic: topicDEBUG Partition: 11 Leader: 7 Replicas: 7,8,1 Isr: 7,8,1
> Topic: topicDEBUG Partition: 12 Leader: 8 Replicas: 8,1,2 Isr: 8,2
> Topic: topicDEBUG Partition: 13 Leader: 3 Replicas: 1,2,3 Isr: 3,2
> Topic: topicDEBUG Partition: 14 Leader: 2 Replicas: 2,3,4 Isr: 3,4,2
> Topic: topicDEBUG Partition: 15 Leader: 3 Replicas: 3,4,5 Isr: 3,4,5
> Topic:topicTRACE PartitionCount:16 ReplicationFactor:3 Configs:
> retention.ms
> =3600000
> Topic: topicTRACE Partition: 0 Leader: 5 Replicas: 5,8,1 Isr: 5,8,1
> Topic: topicTRACE Partition: 1 Leader: 6 Replicas: 6,1,2 Isr: 6,1,2
> Topic: topicTRACE Partition: 2 Leader: 7 Replicas: 7,2,3 Isr: 3,7,2
> Topic: topicTRACE Partition: 3 Leader: 8 Replicas: 8,3,4 Isr: 3,4,8
> Topic: topicTRACE Partition: 4 Leader: 1 Replicas: 1,4,5 Isr: 1,5,4
> Topic: topicTRACE Partition: 5 Leader: 2 Replicas: 2,5,6 Isr: 5,6,2
> Topic: topicTRACE Partition: 6 Leader: 3 Replicas: 3,6,7 Isr: 3,6,7
> Topic: topicTRACE Partition: 7 Leader: 4 Replicas: 4,7,8 Isr: 4,7,8
> Topic: topicTRACE Partition: 8 Leader: 5 Replicas: 5,1,2 Isr: 5,1,2
> Topic: topicTRACE Partition: 9 Leader: 6 Replicas: 6,2,3 Isr: 3,6,2
> Topic: topicTRACE Partition: 10 Leader: 7 Replicas: 7,3,4 Isr: 3,4,7
> Topic: topicTRACE Partition: 11 Leader: 8 Replicas: 8,4,5 Isr: 4,5,8
>
>
> Problem is one of my topic's ISR is not updating and keep failing to be
> preferred replica. more detail, broker 1 for topicDEBUG's ISR is not
> updating.
> And log of broker 1 is absolutely normal and has no error.
>
> This is expected situation? what I have to updating this?
>
>
> Thanks in advance.
>
>
>
>
>
> --
> *Sincerely*
> *,**Bongyeon Kim*
>
> Java Developer & Engineer
> Seoul, Korea
> Mobile: +82-10-9369-1314
> Email: bongyeonkim@gmail.com
> Twitter: http://twitter.com/tigerby
> Facebook: http://facebook.com/tigerby
> Wiki: http://tigerby.com
>