Disk-size aware partitioning

[Vincent Bernardi <vi...@kameleoon.com>]

Hello everyone,
I couldn't find an answer to this on the web so:
If I have a Kafka cluster where nodes have different disk size, is there a
way to have an automatic partitioning aware of these disk sizes (i.e.
allocating 4 partitions to the 4TB node and 1 partition to the 1TB node)? I
know I can partition by hand, but it would be much easier if I could just
declare a max disk space to the partitioner one way or the other.

Thanks in advance,
Vincent Bernardi

Re: Disk-size aware partitioning

[Brett Rann <br...@zendesk.com.INVALID>]

LInkedin's cruise-control  https://github.com/linkedin/cruise-control

has numerous goals, including disk, network, cpu, rack awareness,
leadership distribution etc.

You can have separate disk/network limits per broker (ours are all the same
fwiw)

We use it and it does a stellar job of keeping a cluster balanced.

On Tue, Oct 9, 2018 at 8:11 PM Vincent Bernardi <vi...@kameleoon.com>
wrote:

> Hello everyone,
> I couldn't find an answer to this on the web so:
> If I have a Kafka cluster where nodes have different disk size, is there a
> way to have an automatic partitioning aware of these disk sizes (i.e.
> allocating 4 partitions to the 4TB node and 1 partition to the 1TB node)? I
> know I can partition by hand, but it would be much easier if I could just
> declare a max disk space to the partitioner one way or the other.
>
> Thanks in advance,
> Vincent Bernardi
>