Kafka 0.9.1 - Uneven Partition to Consumer assignment

[Praveen <pr...@gmail.com>]

I have a topic with 16 partitions.

I also have 24 consumer threads (8 per process per box) subscribed to that
same topic. This configuration ensures that there is plenty of room for 1:1
partition to consumer assignment. And some standby consumers to take over
in case the process dies

But during a kafka restart (one of many), I somehow ended up in a state
where 16 partitions where assigned to only 12 consumer threads.

```

6 partitions to 4 consumers on Box 1

5 partitions to 4 consumers on Box 2

5 partitions to 4 consumers on Box 3

```

I believe at least some of the other consumers on each boxes were still
active

As a result, few of the consumers were overloaded resulting in a bigger lag.

I initially thought this was due to kafka rebalancing but that doesn't seem
to be the case. I am using the default RangeAssigner. I don't believe that
anything will change with RoundRobinAssigner since I only have 1 topic.
(Feel free to correct me if I am wrong about the round robin assigner)

How can I ensure that partitions to consumer assignment is always uniform?

Would appreciate if you have any insights.
Thanks,
Praveen