kafka broker with inconsistent data - NotLeaderForPartitionError

[Nishant Kumar <ni...@gmail.com>]

Hi,

We have a 13 node Kafka cluster and each broker has multiple disks and all
topics have replication factor 3.

Broker 6 had a hardware issue and required a complete OS reload (Linux) and
2 disk replacements. Now I installed Kafka again on this node with the same
broker id 6 but started to get an exception from all producers -

*[Error 6] NotLeaderForPartitionError:
ProduceResponsePayload(topic=u'amyTopic', partition=7, error=6, offset=-1)*

I am assuming that since I am using the same broker ID, it (zookeeper?
or controller broker?) is expecting data in the disk which got replaced or
some other meta info that might get wiped out during OS reload. What are
the options I have to *add this node back to the cluster* without much
disturbance to the cluster and without data loss? Should I use a new broker
ID for this node and then repartition the data of every topic as we do
after adding a new node? We have a lot of data (a few hundred TB) in the
cluster and I am trying to avoid huge data movement caused by the
repartition of data and it may choke the entire cluster. Please suggest.