You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Johansson, Olof" <Ol...@thingworx.com> on 2016/08/23 19:19:52 UTC

replica-assignment tool not working when leader not available

Can’t reassign partition of leader (-1) is unavailable.

Replication-factor: 1
Initial brokers: 2 [1001, 1002]
unclean.leader.election.enable=true

Say broker 1002 went down. As expected kafka-topics describe shows
  Topic: my_topic 	Partition: 5   	Leader: -1     	Replicas: 1002    	Isr:

I’m ok with message loss, and don’t want a higher replication-factor (for this topic). Need to fail-over to the healthy broker (i.e. 1001).
Execute the replica-reassignment tool to move all partitions to healthy broker. Output says executing it is successful.

The kafka-topics describe now shows:
  Topic: my_topic  Partition: 5   	Leader: -1     	Replicas: 1001,1002    	Isr:

The unclean leader (i.e. 1001) is not picked as leader. I have to start up another broker with same id as the failed one (i.e. 1002), at which point the replica-reassignment is completed.

Topic: my_topic	Partition: 5   	Leader: 1001   	Replicas: 1001 	Isr: 1001

I see several threads and questions on the internet asking the same thing, but slightly differently. The vague answer is to run the replica-reassignment tool. But I can’t get it to work… Given Kafka’s general awesomeness I assume I’m missing something… Please help.


The log in the healthy Kafka broker shows after the reassignment the following error:
  [2016-08-23 18:43:43,405] ERROR Broker 1001 received LeaderAndIsrRequest with correlation id 350 from controller 1001 epoch 11 for partition [my_topic,5] but cannot become follower since the new leader -1 is unavailable. (state.change.logger)
  [2016-08-23 18:43:16,028] ERROR Controller 1001 epoch 11 initiated state change for partition [my_topic,9] from OfflinePartition to OnlinePartition failed (state.change.logger)