broker never comes back to ISR

["Yu, Libo " <li...@citi.com>]

Hi team,

We have three kafka brokers in a production cluster. We use replication factor 3 for all topics.
We notice quite frequently one broker is not in isr. Sometimes after it is restarted, it
will go back to isr. Sometimes even after it is restarted, it will not go back to isr.

In today's case, after a broker is restarted, this is what we found from the log:

[2013-08-21 08:22:55,524] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
[2013-08-21 08:25:06,621] INFO Closing socket connection to /xxx.xx.xx.xx. (kafka.network.Processor)
[2013-08-21 08:25:06,716] INFO Closing socket connection to / xxx.xx.xx.xx. (kafka.network.Processor)
[2013-08-21 08:27:19,824] INFO Closing socket connection to / xxx.xx.xx.xx. (kafka.network.Processor)
[2013-08-21 08:28:16,711] INFO Closing socket connection to / xxx.xx.xx.xx. (kafka.network.Processor)
[2013-08-21 08:28:17,978] INFO Closing socket connection to / xxx.xx.xx.xx. (kafka.network.Processor)
...
Numerous "Closing socket connection" and nothing else.

Any guidance will be appreciated.

Regards,

Libo