Smoothly expanding log.dirs

[Will Weber <rw...@gmail.com>]

Hey all!

I was wondering how you all might recommend expanding your log.dirs
settings safely.

Specifically, when you add new entries to your log.dirs setting on your
brokers, and execute a rolling restart to have the brokers take the new
configuration, how do you spread out your partitions into the new directory
that was added? I've noticed that this will not happen automatically for
existing partitions(on version 1.1.1).

A relatively straightforward, albeit dangerous solution is to simply remove
all contents from the existing log.dirs defined in the brokers
configuration(including the new one). I'd really rather not do that going
forward if I could avoid it, as it does put additional strain on the
cluster for what I feel like should be possible to do mostly within a
single broker.

I am aware that there is a more manual method available by modifying the
files in the partition directories, outlined for the most part here:
https://youtu.be/qDjEveb8vzk?t=153

If possible, I'm not looking to move these partitions out of the broker
itself, more just have it balance the number of partitions more evenly
across the available log.dirs.

Thanks in advance for any insight!

Best,

- Will