Kafka using 100% IO disk in read, constantly

["ext-gfenollar@eramet-sln.nc" <ex...@eramet-sln.nc>]

Hello,
Since we upgraded from Kafka 0.8 to Kafka0.10.1, our brokers are using 100% of our disk capacity (100% util in iostat), with from 100MB/s to 1000MB/s constant read stats.
It’s been half a week now, and usage is not decreasing. Note that we didn’t experience that before the upgrade.

There’s nothing particularly helpful in the logs (except the fact that we have corrupted index files at startup that kafka recreated corrupted again, but it was already present in Kafka 0.8).
It leads to a fairly high amount of CPU iowait, in all our environments.
Even weirder thing is that the usage is not exactly the same on all brokers. In first one, we have 100MB/s, 200MB/s for the 2nd one, and 1000MB/s for the 3rd one, all of them having the same datastore, same disk speed.

With htop, I can see that only one Java thread is doing that, but I’m not sure how to gather much info about it.

Can you help ?
Thanks,

Guillaume
________________________________
CONFIDENTIALITE
L'information contenue dans ce courrier électronique et ses pièces jointes est confidentielle, et est établie à l'intention exclusive de ses destinataires. Dans le cas où ce message ne vous serait pas destiné, nous vous remercions de bien vouloir en aviser immédiatement l'émetteur et de procéder à sa suppression. Toutes copies, diffusions ou accès non autorisés à ce message sont interdits à toutes personnes, autre que le(s) destinataire(s). Un courrier électronique est susceptible d’altération ou de falsification et peut entrainer des pertes et/ou la destruction de données. Le Groupe ERAMET et/ou ses filiales déclinent toute responsabilité en la matière. En conséquence ce courrier électronique ainsi que ses pièces jointes sont utilisés à votre propre risque.

CONFIDENTIALITY
The information contained in this e-mail and any accompanying documents is confidential or otherwise protected from disclosure. If you are not the intended recipient, please immediately alert the sender by reply e-mail and delete this message and any attachments. Any copy, dissemination or unauthorized access of the contents of this message by anyone other than the intended recipient is strictly prohibited. E-mails may be susceptible to falsification or alteration and cause data corruption and/or loss of data. ERAMET and/or any of its subsidiaries decline any liability resulting from the consequences thereof. Therefore, this e-mail and any attachments are used at your own risk.