You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Sriharsha Chintalapani (JIRA)" <ji...@apache.org> on 2015/02/20 17:58:12 UTC
[jira] [Issue Comment Deleted] (KAFKA-1887) controller error
message on shutting the last broker
[ https://issues.apache.org/jira/browse/KAFKA-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sriharsha Chintalapani updated KAFKA-1887:
------------------------------------------
Comment: was deleted
(was: [~nehanarkhede] Another problem I forgot add in my comment above with this fix i.e moving kafkaHealthCheck below the controller shutdown causing broker controlled shutdown to fail . Here is the log on a 2 node cluster with 2 partitions, 2 replication factor topic.
This happens after I shutdown second broker and next shutdown the controller
I'll get more details on this.
[2015-02-20 08:24:34,548] INFO [Kafka Server 0], Starting controlled shutdown (kafka.server.KafkaServer)
[2015-02-20 08:24:34,569] INFO [Kafka Server 0], Remaining partitions to move: [test3,0],[test3,1] (kafka.server.KafkaServer)
[2015-02-20 08:24:34,570] INFO [Kafka Server 0], Error code from controller: 0 (kafka.server.KafkaServer)
[2015-02-20 08:24:39,575] WARN [Kafka Server 0], Retrying controlled shutdown after the previous attempt failed... (kafka.server.KafkaServer)
[2015-02-20 08:24:39,596] INFO [Kafka Server 0], Remaining partitions to move: [test3,0],[test3,1] (kafka.server.KafkaServer)
[2015-02-20 08:24:39,596] INFO [Kafka Server 0], Error code from controller: 0 (kafka.server.KafkaServer)
^C[2015-02-20 08:24:44,598] WARN [Kafka Server 0], Retrying controlled shutdown after the previous attempt failed... (kafka.server.KafkaServer)
[2015-02-20 08:24:44,617] INFO [Kafka Server 0], Remaining partitions to move: [test3,0],[test3,1] (kafka.server.KafkaServer)
[2015-02-20 08:24:44,617] INFO [Kafka Server 0], Error code from controller: 0 (kafka.server.KafkaServer)
[2015-02-20 08:24:49,620] WARN [Kafka Server 0], Retrying controlled shutdown after the previous attempt failed... (kafka.server.KafkaServer)
[2015-02-20 08:24:49,621] INFO Closing socket connection to /192.168.202.1. (kafka.network.Processor)
[2015-02-20 08:24:49,621] WARN [Kafka Server 0], Proceeding to do an unclean shutdown as all the controlled shutdown attempts failed (kafka.server.KafkaServer)
)
> controller error message on shutting the last broker
> ----------------------------------------------------
>
> Key: KAFKA-1887
> URL: https://issues.apache.org/jira/browse/KAFKA-1887
> Project: Kafka
> Issue Type: Bug
> Components: core
> Reporter: Jun Rao
> Assignee: Sriharsha Chintalapani
> Priority: Minor
> Fix For: 0.8.3
>
>
> We always see the following error in state-change log on shutting down the last broker.
> [2015-01-20 13:21:04,036] ERROR Controller 0 epoch 3 initiated state change for partition [test,0] from OfflinePartition to OnlinePartition failed (state.change.logger)
> kafka.common.NoReplicaOnlineException: No replica for partition [test,0] is alive. Live brokers are: [Set()], Assigned replicas are: [List(0)]
> at kafka.controller.OfflinePartitionLeaderSelector.selectLeader(PartitionLeaderSelector.scala:75)
> at kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:357)
> at kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:206)
> at kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply(PartitionStateMachine.scala:120)
> at kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply(PartitionStateMachine.scala:117)
> at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:117)
> at kafka.controller.KafkaController.onBrokerFailure(KafkaController.scala:446)
> at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:373)
> at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
> at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:358)
> at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
> at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:356)
> at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)