You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "David Dufour (Jira)" <ji...@apache.org> on 2023/06/19 07:43:00 UTC
[jira] [Updated] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Dufour updated KAFKA-15102:
---------------------------------
Description:
According to KIP690, "When users upgrade an existing MM2 cluster they don’t need to change any of their current configuration as this proposal maintains the default behaviour for MM2."
Now, the separator is subject to customization.
As a consequence, when an MM2 upgrade is performed, if the separator was customized with replication.policy.separator, the name of this internal topic changes. It then generates issues like:
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidTopicException: Topic 'mm2-offset-syncs_bkts28_internal' collides with existing topics: mm2-offset-syncs.bkts28.internal
It has been observed that the replication can then be broken sometimes several days after the upgrade (reason not identified). By deleting the old topic name, it recovers.
was:
According to KIP690, "When users upgrade an existing MM2 cluster they don’t need to change any of their current configuration as this proposal maintains the default behaviour for MM2."
In particular, the naming of the topic {{mm2-offset-syncs}} was supposed to use fixed separators:
{{ }}{{default}} {{String offsetSyncsTopic(String targetAlias) { }}{{return}} {{" mm2-offset-syncs."}} {{+ targetCluster + }}{{{}".internal"{}}}{{{};}{}}}
The implementation is actually slightly different:
[https://github.com/apache/kafka/blob/400d39bb0efaa4fdda4c0597a63a1ddd17082291/connect/mirror-client/src/main/java/org/apache/kafka/connect/mirror/DefaultReplicationPolicy.java#L84]
Instead, the separator is subject to customization.
As a consequence, when an upgrade is performed, if the separator is customized with replication.policy.separator, the name of this internal topic changes. It then generates issues like:
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidTopicException: Topic 'mm2-offset-syncs_bkts28_internal' collides with existing topics: mm2-offset-syncs.bkts28.internal
It has been observed that the replication can then be broken sometimes several days after the upgrade (reason not identified). By deleting the old topic name, it recovers.
> Mirror Maker 2 - KIP690 backward compatibility
> ----------------------------------------------
>
> Key: KAFKA-15102
> URL: https://issues.apache.org/jira/browse/KAFKA-15102
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 3.1.0
> Reporter: David Dufour
> Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t need to change any of their current configuration as this proposal maintains the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was customized with replication.policy.separator, the name of this internal topic changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidTopicException: Topic 'mm2-offset-syncs_bkts28_internal' collides with existing topics: mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes several days after the upgrade (reason not identified). By deleting the old topic name, it recovers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)