90 open file handles for one segment

[Matt Kocubinski <mk...@gmail.com>]

A 5 node cluster is running Kafka 1.0.0 with 126 topics, most of them with
24 partitions.  Brokers are keeping around 800k file handles open each, and
regularly crashing with "Too many open file handles".

Using one broker as a study I discovered There are ~13k log file
segments/indices in the Kafka data dir discovered like:

admin@kafka-1:/kafka/logs$ find . -type f | wc -l
12368

 I've also noticed that a single segment file is open in 90 different
threads:

admin@kafka-1:/kafka/logs$ sudo lsof | grep
"/site.time.tracking-6/00000000000000000297.log" | wc -l
90

This is about 1 per broker thread running!  Extrapolating this nearly adds
up to 1.3mm, or the 800k I'm seeing.  Something must be grossly
mis-configured here.  Why are the same segments being opened so many times?