Kafka issue - Error while handling broker changes

[Anthony Pastor <an...@stickyads.tv>]

Hello,

We've an issue on our Kafka production. (Kafka 0.8.1.1 on 9 Debian Wheezy
servers).
We tried to move a topic from a server to another, but no data was created
on the target.

After some tries when decidec to stop Kafka on this server (kafka3),
removed all topic's data on /var/lib/kafka/*
and start again kafka daemon.

Now no data are written at all on kafka3 server.
On kafka1 controller.log we had these logs when whe restarted kafka3's
kafka daemon:

[2015-02-16 11:11:05,822] INFO [BrokerChangeListener on Controller 1]:
Broker change listener fired for path /brokers/ids with children
3,2,1,6,5,4,9,8 (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
[2015-02-16 11:11:05,836] INFO [BrokerChangeListener on Controller 1]:
Newly added brokers: 3, deleted brokers: , all live brokers:
5,1,6,9,2,3,8,4 (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
[2015-02-16 11:11:05,836] DEBUG [Channel manager on controller 1]:
Controller 1 trying to connect to broker 3
(kafka.controller.ControllerChannelManager)
[2015-02-16 11:11:05,836] INFO [Controller-1-to-broker-3-send-thread],
Controller 1 connected to id:3,host:172.16.23.3,port:9092 for sending state
change requests (kafka.controller.RequestSendThread)
[2015-02-16 11:11:05,836] INFO [Controller 1]: New broker startup callback
for 3 (kafka.controller.KafkaController)
[2015-02-16 11:11:05,836] INFO [Controller-1-to-broker-3-send-thread],
Starting  (kafka.controller.RequestSendThread)
[2015-02-16 11:11:05,837] ERROR [BrokerChangeListener on Controller 1]:
Error while handling broker changes
(kafka.controller.ReplicaStateMachine$BrokerChangeListener)
java.lang.IllegalStateException: Controller to broker state change requests
batch is not empty while creating a new one. Some UpdateMetadata state
changes Map(3 -> Map([AdBidderZoneSecondKey,0] ->
(LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:6,ControllerEpoch:5),ReplicationFactor:2),AllReplicas:2,3),
[AdBidderZoneSourceAdApprovalDailyKey,0] ->
(LeaderAndIsrInfo:(Leader:1,ISR:1,2,LeaderEpoch:2,ControllerEpoch:3),ReplicationFactor:2),AllReplicas:2,1),
[AdBidderZoneDealDailyKey,0] ->
(LeaderAndIsrInfo:(Leader:4,ISR:4,5,LeaderEpoch:45,ControllerEpoch:5),ReplicationFactor:2),AllReplicas:4,5),
[Views,2] ->
(LeaderAndIsrInfo:(Leader:1,ISR:1,4,LeaderEpoch:5,ControllerEpoch:4),ReplicationFactor:2),AllReplicas:1,4),
[AdBidderZoneTrackIdsDailyKey,0] ->
(LeaderAndIsrInfo:(Leader:2,ISR:2,LeaderEpoch:7,ControllerEpoch:5),ReplicationFactor:2),AllReplicas:2,3)))
might be lost
    at
kafka.controller.ControllerBrokerRequestBatch.newBatch(ControllerChannelManager.scala:202)
    at
kafka.controller.KafkaController.sendUpdateMetadataRequest(KafkaController.scala:939)
    at
kafka.controller.KafkaController.onBrokerStartup(KafkaController.scala:386)
    at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:342)
    at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:330)
    at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:330)
    at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
    at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:329)
    at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:328)
    at
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:328)
    at kafka.utils.Utils$.inLock(Utils.scala:538)
    at
kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:327)
    at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
    at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)


Any help would be most welcome.

-- 

Anthony Pastor.

<http://www.stickyads.tv/>

[image:
http://www.stickyads.tv/wp-content/uploads/2013/03/StickyADS.tv-Black-Logo-PNG-300x46.png]
<http://www.stickyads.tv/>
PARIS | *MONTPELLIER* | MADRID | MILAN | HAMBURG | LONDON