kafka streams replication.factor=3 triples the count of tasks for internal topics

[Xiyuan Hu <xi...@gmail.com>]

Hi,

I noticed that by default, replication.factor is 1 for Kafka Streams.
I increased it to 3 as the document suggested.
Before, I have 2 repartition topics and each has 63 partitions. After
increasing to replication 3, those each repartition topic has 63 * 3 =
189 tasks.

Is these behavior expected? I though the replication happened on
broker side, instead of letting client process the tripled traffic.

Or is there any other config I missed?
Thanks for the help!

Re: kafka streams replication.factor=3 triples the count of tasks for internal topics

["Matthias J. Sax" <ma...@confluent.io>]

The replication factor should not change the number of create tasks.
Those only depend on the number of partitions (and your topology).

If you have two repartition topics I assume that your topology has 3
sub-topologies.

Hence, 2 repartition topic with 68 partitons each (and I assume an input
topic with 68 partitions will result in 189 tasks, 68 task per
sub-topology. Even with a replication factor of one, you should have 189
tasks.


-Matthias



On 11/5/19 2:45 PM, Xiyuan Hu wrote:
> Hi,
> 
> I noticed that by default, replication.factor is 1 for Kafka Streams.
> I increased it to 3 as the document suggested.
> Before, I have 2 repartition topics and each has 63 partitions. After
> increasing to replication 3, those each repartition topic has 63 * 3 =
> 189 tasks.
> 
> Is these behavior expected? I though the replication happened on
> broker side, instead of letting client process the tripled traffic.
> 
> Or is there any other config I missed?
> Thanks for the help!
>