[jira] [Commented] (KAFKA-16344) Internal topic mm2-offset-syncsinternal created with single partition is putting more load on the broker

["Greg Harris (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-16344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823817#comment-17823817 ] 

Greg Harris commented on KAFKA-16344:
-------------------------------------

Hi [~janardhanag], thanks for the ticket.

At the current time, the offset syncs topic cannot have more than 1 partition. If more than one partition is present, the MirrorSourceTask will only write to partition 0, and the MirrorCheckpointTask will only read from partition 0. Changes to both of these would be necessary to support partitioning that topic, and would require a KIP.

For a workaround, are you able to increase your `offset.lag.max` from the default 100? This will make the offset translation less accurate, but can decrease the throughput on that topic.

Do you have some more statistics that confirm that this topic is too active? It should have just a fraction of the load of the other topics, so i'd be interested to know more about the scale you're working at.

Thanks,
Greg

> Internal topic mm2-offset-syncs<clustername>internal created with single partition is putting more load on the broker
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-16344
>                 URL: https://issues.apache.org/jira/browse/KAFKA-16344
>             Project: Kafka
>          Issue Type: Bug
>          Components: connect
>    Affects Versions: 3.5.1
>            Reporter: Janardhana Gopalachar
>            Priority: Major
>
> We are using Kafka 3.5.1 version, we see that the internal topic created by mirrormaker 
> mm2-offset-syncs<clustername>internal is created with single partition due to which the CPU load on the broker which will be leader for this partition is increased compared to other brokers. Can multiple partitions be  created for the topic so that the CPU load would get distributed 
>  
> Topic: mm2-offset-syncscluster-ainternal    TopicId: XRvTDbogT8ytNhqX2YTyrA    PartitionCount: 1ReplicationFactor: 3    Configs: min.insync.replicas=2,cleanup.policy=compact,message.format.version=3.0-IV1
>     Topic: mm2-offset-syncscluster-ainternal    Partition: 0    Leader: 2    Replicas: 2,1,0    Isr: 2,1,0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)