Clustered Kafka/Zookeeper setup not rebalancing all partitions

[Karl Kirch <kk...@wdtinc.com>]

Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions per node) and Zookeeper 3.3.4 on each node. When we first start up the cluster, there are no issues and things work just fine, but after a while things start acting a bit haywire. It looks like we start to miss messages that should be coming through the system. However on client restart, logging shows that only the partitions on broker 2 and 3 are being re-balanced (for the particular topic I'm connecting to). I'm confidant that there is nothing else connected to the cluster (as we've seen this a couple of times now). To make it more interesting, if I switch topics, the client will rebalance the partitions on all three brokers.

Anyone know of any reason why this could occur? 

Thanks,
Karl

Re: Clustered Kafka/Zookeeper setup not rebalancing all partitions

[Jun Rao <ju...@gmail.com>]

Could you run the tool kafka.tools.ConsumerOffsetChecker? It will show you
which consumer instance owns which partition.

Thanks,

Jun


On Tue, Apr 23, 2013 at 12:12 AM, Karl Kirch <kk...@wdtinc.com> wrote:

> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10
> partitions per node) and Zookeeper 3.3.4 on each node. When we first start
> up the cluster, there are no issues and things work just fine, but after a
> while things start acting a bit haywire. It looks like we start to miss
> messages that should be coming through the system. However on client
> restart, logging shows that only the partitions on broker 2 and 3 are being
> re-balanced (for the particular topic I'm connecting to). I'm confidant
> that there is nothing else connected to the cluster (as we've seen this a
> couple of times now). To make it more interesting, if I switch topics, the
> client will rebalance the partitions on all three brokers.
>
> Anyone know of any reason why this could occur?
>
> Thanks,
> Karl