Consumers constantly rebalancing

[Chhavi Parasher <ch...@zapr.in>]

Hi all,

I am using kafka* 0.8.2* broker and a *high level consumer API.*
*Topic properties:*
Number of partitions = 150
Replication factor = 2
Brokers count in cluster = 4

*High level Consumer properties: *
There is only one consumer group for this topic.
I am running 150 parallel consumer threads equally distributed among 15
nodes,
zookeeper.session.timeout.ms=60000
rebalance.backoff.ms = 60000
zookeeper.sync.time.ms = 10000
auto.commit.interval.ms = 60000
rebalance.max.retries = 10
auto.offset.reset = largest

The problem here is I am seeing consumers rebalancing very frequently (less
than a minute). I tried to increase the zookeeper sync.time, max.retries
and session.timeout but none of them solved the issue.
Can anyone help me with this problem? What are the possible causes of
consumer rebalancing? None of my consumer programs are crashing, they are
long running java programs.


Thank you


*Chhavi Parasher *

SeNIOR
SOFTWARE ENGINEER

Zapr Media Labs, 1st Floor, 621/B 12th Main Road, HAL II Stage,

Indiranagar, Bangalore - 560038

 +91-9663444755   chhavi@zapr.in   www.zapr.in