Recovery skipped after unclean shutdown

[Pradeep Jawahar <pj...@groupon.com>]

One of the brokers in our cluster had an unclean shutdown and after it was
restated I found the following logs.

$ grep "clean shutdown" /var/groupon/kafka/kafka-broker.log
02/Sep/2015 16:19:23   - warn::[Kafka Server 1], Proceeding to do an
unclean shutdown as all the controlled shutdown attempts failed
02/Sep/2015 16:22:06   - info::Found clean shutdown file. Skipping recovery
for all logs in data directory '/data/vol1'
02/Sep/2015 16:22:11   - info::Found clean shutdown file. Skipping recovery
for all logs in data directory '/data/vol2'
02/Sep/2015 16:22:15   - info::Found clean shutdown file. Skipping recovery
for all logs in data directory '/data/vol3'
02/Sep/2015 16:22:18   - info::Found clean shutdown file. Skipping recovery
for all logs in data directory '/data/vol4'
02/Sep/2015 16:22:22   - info::Found clean shutdown file. Skipping recovery
for all logs in data directory '/data/vol5'
02/Sep/2015 16:22:22   - info::Found clean shutdown file. Skipping recovery
for all logs in data directory '/data/vol6'
02/Sep/2015 16:22:26   - info::Found clean shutdown file. Skipping recovery
for all logs in data directory '/data/vol7'
02/Sep/2015 16:22:29   - info::Found clean shutdown file. Skipping recovery
for all logs in data directory '/data/vol8'

So no recovery happened and the partitions managed by this broker are not
catching up with the other replicas. I found that the ReplicaFetcher
threads for each of the partitions died.

Is anyone aware of how to get out of this situation. I was trying to locate
the shutdown file (may be it was left over from a previous run) and delete
it.


Additional Information
~~~~~~~~~~~~~~~~~
Kafka v 0.8.1.1
Centos 5
11 node cluster with replication factor 3
Disks are JBODs

Thanks,
Pradeep