You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Рябков Алексей Николаевич <a....@ntc-vulkan.ru> on 2014/05/26 16:06:36 UTC

multi-broker topic rebalance

Hello!

How can I rebalance topic within multiple brokers?
For example:

1.        I have about 30k consumers...

2.        I need to use 3 broker (10k consumer per broker)

3.       I use multiple (around 10) producers with default balance strategy
Sometimes one broker have empty topic (consumer  finished all message on broker)...But another broker have a lot of messages in topic...

How to resolve such situation:
-i cant just switch my consumers to another broker ( broker can not handle more than 10k consumers....)
-I can not find how to rebalance topic within multiple brokers...
- I can not find how to setup limit for message per topic (or partition) ... If such things exists I can handle such case when add new message to broker in consumer side...and switch to  another ...
Thanks in advance, Aleksey Ryabkov