[jira] [Created] (KAFKA-14807) MirrorMaker2 config source.consumer.auto.offset.reset=latest leading to the pause of replication of consumer groups

["Zhaoli (Jira)" <ji...@apache.org>]

Zhaoli created KAFKA-14807:
------------------------------

             Summary: MirrorMaker2 config source.consumer.auto.offset.reset=latest leading to the pause of replication of consumer groups
                 Key: KAFKA-14807
                 URL: https://issues.apache.org/jira/browse/KAFKA-14807
             Project: Kafka
          Issue Type: Bug
          Components: mirrormaker
    Affects Versions: 3.3.2, 3.3.1, 3.4.0
         Environment: centos7
            Reporter: Zhaoli


We use MirrorMaker2 to replicate messages and consumergroup offsets from kafka cluster `source` to cluster `target`.
In order to reduce the load on the source cluster, we add this configuration to mm2 to avoid replicate the whole history messages:
{code:java}
source.consumer.auto.offset.reset=latest {code}
After that, we found part of the consumergroup offsets have stopped replicating.

The common characteristic of these consumergroups is  their EMPTY status，which means they have no active members at that monent. All the active consumergroups‘ offset replication work as normal.

After researching the source code，we found this is because the configuration above also affect the consumption of topic `mm2-offset-syncs`, therefore the map `
offsetSyncs` dosen't hold the whole topicPartitions, and the lost topicPartitions lead to the pause of replication of the EMPTY consumer groups.
{code:java}
private final Map<TopicPartition, OffsetSync> offsetSyncs = new HashMap<>(); {code}
 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)