NotLeaderForPartitionException -- how to recover?

[Samuel Taylor <sg...@samueltaylor.org>]

Hi everyone,

Has anyone seen this error and/or is there a good way to recover from it?

I have two brokers running, and after attempting to produce to a topic
(using the library kafka-python), and they are both spitting out lots of
messages that look like this:

ERROR [ReplicaFetcherThread-0-1], Error for partition
[__consumer_offsets,2] to broker 1:org.apache.kafka.common.errors.
NotLeaderForPartitionException: This server is not the leader for that
topic-partition. (kafka.server.ReplicaFetcherThread)
(The partition number varies. And on broker 1, the message says "to broker
2" instead of "to broker 1")

Listing the topics reveals that broker 1 is the leader for all 50
partitions of __consumer_offsets. Both brokers are complaining about the
same partitions, which is confusing to me because it looks like broker 1
should be the leader.

I have tried:
- restarting Kafka on each broker
- deleting the kafka-logs directory on each broker (and restarting)
- deleting the zookeeper directory on each server (and restarting)

Very grateful for any help!

Thanks,
Samuel