How long should it take to shut down a small cluster?

[Skip Montanaro <sk...@gmail.com>]

For dev purposes, I run a small Kafka/Zookeeper setup, two brokers,
one Zookeeper, all running on the same host. I wrote a small stop
script which executes kafka-server-stop.sh on the two brokers, waits
up to 20 seconds, checking every second to see if the brokers have
both exited, then runs zookeeper-server-stop.sh.

The Zookeeper instance and one of the brokers never seems to exit.
Before running the stop script, I stopped all my clients. I've
confirmed with lsof that the only established IPv4 connections either
has is with the other. I'm running from a kafka_2.11-1.0.0 download on
a Red Hat 6 server.

I don't routinely kill this setup, but when I do, it never seems to
exit cleanly. I always wind up having to forcibly kill the remaining
two processes. Am I doing something wrong? How long should shutdown
typically take?

Thanks,

Skip Montanaro