source topic partitions not assigned evenly to kafka stream threads

[Pushkar Deole <pd...@gmail.com>]

Hi,

I have a microservice hosting kafka streams application. I have 3 instances
of microservice hosted in 3 pods, each is having 2 kafka stream threads,
thus total 6 stream threads as part of consumer group.
There are 3 source topics: A, B, C each having 12, 6, 6 partitions
respectively i.e. total 24 source topic partitions.

Now, the issue I am facing is the distribution of source topic partitions
among stream threads. Considering that 6 streams threads and overall 24
topic partitions, each stream thread is assigned 4 partitions, so no issue
there. However the main issue arises is that of partitions assigned to
stream thread from each topic. So, sometimes, I am getting 4 partitions
from Topic A assigned to stream thread 1, while other stream thread will
get partitions from Topic B and C only.

What I am expecting is, partitions from each topic will get evenly
distributed among 6 threads, so each stream thread should get 2 partitions
of topic A, and 1 partition of topic B and C each, thus making 4 partitions
of input topics. This is required for proper load balancing since topic A
carries a lot more event traffic than topic B and C due to which partitions
of each topic should be distributed evenly across all stream threads.

I am using kafka stream 2.5.1
Is this the default behavior of kafka streams 2.5.1 client? Is there any
configuration to change this behaviour?
Is there any change in logic in later versions of streams that would
guarantee the distribution the way I described above?

Re: source topic partitions not assigned evenly to kafka stream threads

[Pushkar Deole <pd...@gmail.com>]

@Matthias

Can you help with this since I remember having conversation with you in the
past on this topic wherein it was mentioned that partition assignment to
stream task logic might change in future releases

On Mon, Jun 13, 2022, 11:05 Pushkar Deole <pd...@gmail.com> wrote:

> Hi,
>
> I have a microservice hosting kafka streams application. I have 3
> instances of microservice hosted in 3 pods, each is having 2 kafka stream
> threads, thus total 6 stream threads as part of consumer group.
> There are 3 source topics: A, B, C each having 12, 6, 6 partitions
> respectively i.e. total 24 source topic partitions.
>
> Now, the issue I am facing is the distribution of source topic partitions
> among stream threads. Considering that 6 streams threads and overall 24
> topic partitions, each stream thread is assigned 4 partitions, so no issue
> there. However the main issue arises is that of partitions assigned to
> stream thread from each topic. So, sometimes, I am getting 4 partitions
> from Topic A assigned to stream thread 1, while other stream thread will
> get partitions from Topic B and C only.
>
> What I am expecting is, partitions from each topic will get evenly
> distributed among 6 threads, so each stream thread should get 2 partitions
> of topic A, and 1 partition of topic B and C each, thus making 4 partitions
> of input topics. This is required for proper load balancing since topic A
> carries a lot more event traffic than topic B and C due to which partitions
> of each topic should be distributed evenly across all stream threads.
>
> I am using kafka stream 2.5.1
> Is this the default behavior of kafka streams 2.5.1 client? Is there any
> configuration to change this behaviour?
> Is there any change in logic in later versions of streams that would
> guarantee the distribution the way I described above?
>