[jira] [Created] (KAFKA-18125) Implement MirrorCheckpointConnector max task limit

["Jarkko Jaakola (Jira)" <ji...@apache.org>]

Jarkko Jaakola created KAFKA-18125:
--------------------------------------

             Summary: Implement MirrorCheckpointConnector max task limit
                 Key: KAFKA-18125
                 URL: https://issues.apache.org/jira/browse/KAFKA-18125
             Project: Kafka
          Issue Type: Improvement
          Components: mirrormaker
            Reporter: Jarkko Jaakola


Mirrormaker 2 MirrorCheckpointConnector currently can create as many tasks as globally configured. Primary limiting factor for number of tasks up to the configured maximum is the number of known consumer groups. On larger mirrored environments where known consumers groups is large this leads to excessive amount of checkpoint tasks.

As the offset sync topic is single partition topic there will be excessive amount of tasks reading from and writing to this topic. The targeted Kafka broker can and will be under heavy burden.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)