How to stop partition rebalancing

[Gopinath <go...@gmail.com>]

Hi,

I have 2 partition in a topic and 3 pods/instances of my microservice
running in my k8s cluster. I wanted all 3 pods to pull message from 2
partitions. (I'm using same group id for all 3 pods/instances).

I have achieved this by closing consumer as soon as i pulled message from
partition. kafkaconsumer.close(); in this way i see i could see 3 pods can
able to pull message from this 2 partition.

But  what i observed is, when ever i close suing kafkaconsumer.close(); i
see rebalancing is happening in partition. I do not want rebalancing to
happen as another pod is ready to pick message from this partition.

Can you please suggest a way to stop rebalancing.

Re: How to stop partition rebalancing

["M. Manna" <ma...@gmail.com>]

Hi,

If you choose not to close consumer, but only poll as a background thread
it, the rebalance wouldn’t take place. This is only if you are sending
heartbeats (via Poll() ) within the timeout period. Also, if one of the pod
dies, rebalance has to happen. You’d want a fare share of work.

An example code is also provided on KafkaConsumer docs. You may want to
check that.

The other option is to use manual partition assignment, but that means you
need to manually assign and manage who gets the partition.

Thanks,


On Wed, 9 Oct 2019 at 20:21, Gopinath <go...@gmail.com> wrote:

> Hi,
>
> I have 2 partition in a topic and 3 pods/instances of my microservice
> running in my k8s cluster. I wanted all 3 pods to pull message from 2
> partitions. (I'm using same group id for all 3 pods/instances).
>
> I have achieved this by closing consumer as soon as i pulled message from
> partition. kafkaconsumer.close(); in this way i see i could see 3 pods can
> able to pull message from this 2 partition.
>
> But  what i observed is, when ever i close suing kafkaconsumer.close(); i
> see rebalancing is happening in partition. I do not want rebalancing to
> happen as another pod is ready to pick message from this partition.
>
> Can you please suggest a way to stop rebalancing.
>

Re: How to stop partition rebalancing

["Matthias J. Sax" <ma...@confluent.io>]

The pattern you apply seems to be an anti-pattern.

If you have only 2 partitions, only 2 consumer instances within a group
can consumer data -- hence, you should either only use 2 pods, or you
should increase the number of partitions.

What you try to do atm, won't work smoothly no matter what you do.


-Matthias

On 10/9/19 11:05 AM, Gopinath wrote:
> Hi,
> 
> I have 2 partition in a topic and 3 pods/instances of my microservice
> running in my k8s cluster. I wanted all 3 pods to pull message from 2
> partitions. (I'm using same group id for all 3 pods/instances).
> 
> I have achieved this by closing consumer as soon as i pulled message from
> partition. kafkaconsumer.close(); in this way i see i could see 3 pods can
> able to pull message from this 2 partition.
> 
> But  what i observed is, when ever i close suing kafkaconsumer.close(); i
> see rebalancing is happening in partition. I do not want rebalancing to
> happen as another pod is ready to pick message from this partition.
> 
> Can you please suggest a way to stop rebalancing.
>