Brokers are failing to handle UpdateMetadata and LeaderAndIsRequest requests from controller

[Chris Richardson <ch...@chrisrichardson.net>]

Hi,

I have a Kafka 1.1 cluster that doesn't start up correctly: many partitions
are not replicated correctly.

In the controller.log I see messages indicating that brokers aren't
succesfully handling LeaderAndIsRequest and UpdateMetadata requests.

For example:

[2018-07-04 13:31:18,788] WARN [RequestSendThread controllerId=6]
Controller 6 epoch 96 fails to send request (type=LeaderAndIsRequest, ....
) ... Reconnecting to broker. (kafka.controller.RequestSendThread)
...
java.io.IOException: Connection to 6 was disconnected before the response
was read



Quite often the controller is sending the LeaderAndIsRequest and
UpdateMetadata requests to itself.
There are a large number of topics/partitions and so the requests are quite
large.

Any suggestions about the cause and solution?

Thanks.

Chris