[jira] [Updated] (KAFKA-13397) Honor 'replication.policy.separator' configuration when creating MirrorMaker2 internal topics

["Dongjin Lee (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-13397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjin Lee updated KAFKA-13397:
--------------------------------
    Description: 
This issue is a follow-up of KAFKA-10777 ([KIP-690|https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention]).

As of present, a user can set custom 'replication.policy.separator' configuration in MirrorMaker 2. It determines the topic name of internal topics like heartbeats, checkpoints, and offset-syncs ([KIP-690|https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention]).

However, there are some glitches here:
 # MirrorMaker2 creates internal topics to track the offsets, configs, and status of the MM2 tasks. But, these topics are not affected by a custom 'replication.policy.separator' settings - that is, these topics may be replicated against the user`s intention.
 # The internal topic names include a dash in their name (e.g., 'mm2-offsets.\{source}.internal') so, a single '-' should be disallowed when configuring 'replication.policy.separator'.

  was:
This issue is a follow-up of KAFKA-10777 ([KIP-690|https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention]) and KAFKA-12379 ([KIP-716|https://cwiki.apache.org/confluence/display/KAFKA/KIP-716%3A+Allow+configuring+the+location+of+the+offset-syncs+topic+with+MirrorMaker2]).

As of present, a user can set custom 'replication.policy.separator' configuration in MirrorMaker 2. It determines the topic name of internal topics like heartbeats, checkpoints, and offset-syncs ([KIP-690|https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention]). Also, the user can configure on which side the offset-syncs topic will be created ([KIP-716|https://cwiki.apache.org/confluence/display/KAFKA/KIP-716%3A+Allow+configuring+the+location+of+the+offset-syncs+topic+with+MirrorMaker2]) with 'offset-syncs.topic.location'.

However, there are some glitches here:
 # MirrorMaker2 creates internal topics to track the offsets, configs, and status of the MM2 tasks. But, these topics are not affected by a custom 'replication.policy.separator' settings - that is, these topics may be replicated against the user`s intention.
 # The internal topic names include a dash in their name (e.g., 'mm2-offsets.\{source}.internal') so, a single '-' should be disallowed when configuring 'replication.policy.separator'. 
 # By default, the offset-syncs topic is created in the source cluster and also follows the 'replication.policy.separator' configuration of the source side. But if the user changes the location of the offset-syncs topic to target, it still follows the source's 'replication.policy.separator' configuration, not target one.

These issues seem like a corner case between the two above KIPs.


> Honor 'replication.policy.separator' configuration when creating MirrorMaker2 internal topics
> ---------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-13397
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13397
>             Project: Kafka
>          Issue Type: Bug
>          Components: mirrormaker
>            Reporter: Dongjin Lee
>            Assignee: Dongjin Lee
>            Priority: Minor
>
> This issue is a follow-up of KAFKA-10777 ([KIP-690|https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention]).
> As of present, a user can set custom 'replication.policy.separator' configuration in MirrorMaker 2. It determines the topic name of internal topics like heartbeats, checkpoints, and offset-syncs ([KIP-690|https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention]).
> However, there are some glitches here:
>  # MirrorMaker2 creates internal topics to track the offsets, configs, and status of the MM2 tasks. But, these topics are not affected by a custom 'replication.policy.separator' settings - that is, these topics may be replicated against the user`s intention.
>  # The internal topic names include a dash in their name (e.g., 'mm2-offsets.\{source}.internal') so, a single '-' should be disallowed when configuring 'replication.policy.separator'.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)