[jira] [Updated] (KAFKA-9914) Mirror Maker 2 creates heartbeats kafka topics recursively

["azher khan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

azher khan updated KAFKA-9914:
------------------------------
    Description: 
Hi Team,

I configured Mirror Maker 2 to run on a Pod configuring the Kafka brokers in our environments marking '10.X.X.YY' Kafka broker as primary and '10.X.Y.XY' as backup.

+*Mirror Maker mm2.properties file:*+
{code:java}
clusters=kafka,backup
kafka.bootstrap.servers=10.X.X.YY:9092
backup.bootstrap.servers=10.X.Y.XY:9092
kafka->backup.enabled=true
kafka->backup.topics=az.*
backup->kafka.enabled=false
sync.topic.acls.enabled=true
{code}
 

 

I was able to run Mirror Maker 2 successfully and was also able to take backup of the identified topics (starting with 'az').

 

However I could see many kafka topics with the suffix 'heartbeats' created in recursion (See below the list of topics with 'kafka.*.heartbeats'). This could be because i triggered the 'connect-mirror-maker.sh' several times to test backing up other topics

 

I have 2 queries
 # How to avoid having 'kafka.*.heartbeats' topic being created using Mirror Maker 2
 # Once Mirror Maker 2 has backed up a topic say 'azherf1test' . What is the best way to rollback changes made by Mirror Maker (ie delete all the topics created by Mirror Maker 'kafka.azherf1test' and supporting topics) ensuring the stability of existing/source topics and Kafka broker.

We are testing Mirror Maker and want to ensure we are able to roll back the changes without affecting the Kafka topics/brokers)

+*Kafka Topics list output:*+
{code:java}
azherf1test
 heartbeats
 kafka-client-topic
 mm2-configs.backup.internal
 mm2-configs.kafka.internal
 mm2-offset-syncs.backup.internal
 mm2-offsets.backup.internal
 mm2-offsets.kafka.internal
 mm2-status.backup.internal
 mm2-status.kafka.internal
 kafka.azherf1test
 kafka.checkpoints.internal
 kafka.heartbeats
 kafka.kafka.heartbeats
 kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats{code}

 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
 kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats

  was:
Hi Team,

I configured Mirror Maker 2 to run on a Pod configuring the Kafka brokers in our environments marking '10.X.X.YY' Kafka broker as primary and '10.X.Y.XY' as backup.

// code placeholder
clusters=kafka,backup

kafka.bootstrap.servers=10.X.X.YY:9092

backup.bootstrap.servers=10.X.Y.XY:9092

kafka->backup.enabled=true

kafka->backup.topics=az.*

backup->kafka.enabled=false

sync.topic.acls.enabled=true

 

I was able to run Mirror Maker 2 successfully and was also able to take backup of the identified topics (starting with 'az').

 

However I could see many kafka topics with the suffix 'heartbeats' created in recursion (See below the list of topics with 'kafka.*.heartbeats'). This could be because i triggered the 'connect-mirror-maker.sh' several times to test backing up other topics

 

I have 2 queries
 # How to avoid having 'kafka.*.heartbeats' topic being created using Mirror Maker 2
 # Once Mirror Maker 2 has backed up a topic say 'azherf1test' . What is the best way to rollback changes made by Mirror Maker (ie delete all the topics created by Mirror Maker 'kafka.azherf1test' and supporting topics) ensuring the stability of existing/source topics and Kafka broker.

We are testing Mirror Maker and want to ensure we are able to roll back the changes without affecting the Kafka topics/brokers)


// code placeholder
azherf1test
heartbeats
kafka-client-topic
mm2-configs.backup.internal
mm2-configs.kafka.internal
mm2-offset-syncs.backup.internal
mm2-offsets.backup.internal
mm2-offsets.kafka.internal
mm2-status.backup.internal
mm2-status.kafka.internal
kafka.azherf1test
kafka.checkpoints.internal
kafka.heartbeats
kafka.kafka.heartbeats
kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats


> Mirror Maker 2 creates heartbeats kafka topics recursively
> ----------------------------------------------------------
>
>                 Key: KAFKA-9914
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9914
>             Project: Kafka
>          Issue Type: Bug
>          Components: mirrormaker
>    Affects Versions: 2.0.0
>            Reporter: azher khan
>            Priority: Major
>
> Hi Team,
> I configured Mirror Maker 2 to run on a Pod configuring the Kafka brokers in our environments marking '10.X.X.YY' Kafka broker as primary and '10.X.Y.XY' as backup.
> +*Mirror Maker mm2.properties file:*+
> {code:java}
> clusters=kafka,backup
> kafka.bootstrap.servers=10.X.X.YY:9092
> backup.bootstrap.servers=10.X.Y.XY:9092
> kafka->backup.enabled=true
> kafka->backup.topics=az.*
> backup->kafka.enabled=false
> sync.topic.acls.enabled=true
> {code}
>  
>  
> I was able to run Mirror Maker 2 successfully and was also able to take backup of the identified topics (starting with 'az').
>  
> However I could see many kafka topics with the suffix 'heartbeats' created in recursion (See below the list of topics with 'kafka.*.heartbeats'). This could be because i triggered the 'connect-mirror-maker.sh' several times to test backing up other topics
>  
> I have 2 queries
>  # How to avoid having 'kafka.*.heartbeats' topic being created using Mirror Maker 2
>  # Once Mirror Maker 2 has backed up a topic say 'azherf1test' . What is the best way to rollback changes made by Mirror Maker (ie delete all the topics created by Mirror Maker 'kafka.azherf1test' and supporting topics) ensuring the stability of existing/source topics and Kafka broker.
> We are testing Mirror Maker and want to ensure we are able to roll back the changes without affecting the Kafka topics/brokers)
> +*Kafka Topics list output:*+
> {code:java}
> azherf1test
>  heartbeats
>  kafka-client-topic
>  mm2-configs.backup.internal
>  mm2-configs.kafka.internal
>  mm2-offset-syncs.backup.internal
>  mm2-offsets.backup.internal
>  mm2-offsets.kafka.internal
>  mm2-status.backup.internal
>  mm2-status.kafka.internal
>  kafka.azherf1test
>  kafka.checkpoints.internal
>  kafka.heartbeats
>  kafka.kafka.heartbeats
>  kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats{code}
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats
>  kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.kafka.heartbeats



--
This message was sent by Atlassian Jira
(v8.3.4#803005)