[jira] [Commented] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility

["Chris Egerton (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738733#comment-17738733 ] 

Chris Egerton commented on KAFKA-15102:
---------------------------------------

[~ddufour1a] Thanks for raising this. I believe if we wanted to preserve backward compatibility perfectly, we would have had to ignore the custom separator when creating topics affected by KIP-690, and possibly introduced separate opt-in configuration logic to disable that behavior (i.e., resume taking custom separators into account, even for topics that were not affected prior to KIP-690).

However, since 3.1.0 came out a year and a half ago, things are less cut-and-dry: there may be more negative fallout for users who are accustomed to the current behavior than what's currently being experienced by those who are used to the previous behavior.

One possibility could be to revert to older behavior for the remainder of our 3.x.y releases with a single configuration property such as {{replication.policy.internal.topic.separator.enabled}} that defaults to {{false}} for now and, come 4.0, defaults to {{{}true{}}}.

[~omnia_h_ibrahim] [~mimaison] do either of you have thoughts on how to approach this accidental break in compatibility?

> Mirror Maker 2 - KIP690 backward compatibility
> ----------------------------------------------
>
>                 Key: KAFKA-15102
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15102
>             Project: Kafka
>          Issue Type: Bug
>          Components: mirrormaker
>    Affects Versions: 3.1.0
>            Reporter: David Dufour
>            Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t need to change any of their current configuration as this proposal maintains the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was customized with replication.policy.separator, the name of this internal topic changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidTopicException: Topic 'mm2-offset-syncs_bkts28_internal' collides with existing topics: mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes several days after the upgrade (reason not identified). By deleting the old topic name, it recovers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)