You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Daan Bosch (Jira)" <ji...@apache.org> on 2021/06/11 13:11:00 UTC

[jira] [Updated] (KAFKA-12937) Mirrormaker2 can only start from the beginning of a topic

     [ https://issues.apache.org/jira/browse/KAFKA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daan Bosch updated KAFKA-12937:
-------------------------------
    Description: 
*Goal*:
 I want to replace Mirrormaker version 1 with Mirrormaker2.
 To do this I want to:
 start Mirrormaker2 from the latest offset of every topic
 stop Mirrormaker1 
 There should only be a couple of double messages.

What happened:
 Mirrormaker2 starts replicating from the start of all topics

*How to reproduce:*
 Start two Kafka clusters, A and B

I produce 3000 messages to cluster A on a topic (TOPIC1)
 Kafka Connect is running and connected to cluster B
 Start a Mirrormaker2 task in connect to replicate messages from cluster A. Wit the option:
 consumer auto.offset.reset to latest
 Produce another 3000 messages to cluster A on the same topic (TOPIC1)

*Expected result:*
 Cluster B will only contain the messages produced the second time (3000 in total) on TOPIC1

Actual result:
 The mirror picks up all messages from the start (6000 in total) and replicates them to cluster B

*Additional logs:*
 Logs from the consumer of the Mirrormaker task:

mirrormaker.log:7581:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-8 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
 mirrormaker.log:7583:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-3 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
 mirrormaker.log:7585:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-2 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
 mirrormaker.log:7587:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-1 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
 mirrormaker.log:7589:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
 mirrormaker.log:7591:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-7 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
 mirrormaker.log:7593:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-6 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
 mirrormaker.log:7595:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-5 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
 mirrormaker.log:7597:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-4 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)You can see they are trying to seek to a position and thus overriding the latest offset

 

You can ssee it is doing a seek to position 0 for every partition. which is not what I expected

  was:
*Goal*:
I want to replace Mirrormaker version 1 with Mirrormaker2.
To do this I want to:
start Mirrormaker2 from the latest offset of every topic
stop Mirrormaker1 
There should only be a couple of double messages.

What happened:
Mirrormaker2 starts replicating from the start of all topics


*How to reproduce:*
Start two Kafka clusters, A and B

I produce 3000 messages to cluster A on a topic (TOPIC1)
Kafka Connect is running and connected to cluster B
Start a Mirrormaker2 task in connect to replicate messages from cluster A. Wit the option:
consumer auto.offset.reset to latest
Produce another 3000 messages to cluster A on the same topic (TOPIC1)


*Expected result:*
Cluster B will only contain the messages produced the second time (3000 in total) on TOPIC1

Actual result:
The mirror picks up all messages from the start (6000 in total) and replicates them to cluster B

*Additional logs:*
Logs from the consumer of the Mirrormaker task:

mirrormaker.log:7581:mirrormaker_1                    | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-8 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7583:mirrormaker_1                    | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-3 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7585:mirrormaker_1                    | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-2 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7587:mirrormaker_1                    | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-1 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7589:mirrormaker_1                    | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7591:mirrormaker_1                    | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-7 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7593:mirrormaker_1                    | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-6 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7595:mirrormaker_1                    | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-5 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7597:mirrormaker_1                    | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-4 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)You can see they are trying to seek to a position and thus overriding the latest offset


> Mirrormaker2  can only start from the beginning of a topic
> ----------------------------------------------------------
>
>                 Key: KAFKA-12937
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12937
>             Project: Kafka
>          Issue Type: Bug
>          Components: mirrormaker
>    Affects Versions: 2.8.0
>         Environment: Dockerized environment
>            Reporter: Daan Bosch
>            Priority: Major
>
> *Goal*:
>  I want to replace Mirrormaker version 1 with Mirrormaker2.
>  To do this I want to:
>  start Mirrormaker2 from the latest offset of every topic
>  stop Mirrormaker1 
>  There should only be a couple of double messages.
> What happened:
>  Mirrormaker2 starts replicating from the start of all topics
> *How to reproduce:*
>  Start two Kafka clusters, A and B
> I produce 3000 messages to cluster A on a topic (TOPIC1)
>  Kafka Connect is running and connected to cluster B
>  Start a Mirrormaker2 task in connect to replicate messages from cluster A. Wit the option:
>  consumer auto.offset.reset to latest
>  Produce another 3000 messages to cluster A on the same topic (TOPIC1)
> *Expected result:*
>  Cluster B will only contain the messages produced the second time (3000 in total) on TOPIC1
> Actual result:
>  The mirror picks up all messages from the start (6000 in total) and replicates them to cluster B
> *Additional logs:*
>  Logs from the consumer of the Mirrormaker task:
> mirrormaker.log:7581:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-8 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
>  mirrormaker.log:7583:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-3 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
>  mirrormaker.log:7585:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-2 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
>  mirrormaker.log:7587:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-1 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
>  mirrormaker.log:7589:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
>  mirrormaker.log:7591:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-7 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
>  mirrormaker.log:7593:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-6 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
>  mirrormaker.log:7595:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-5 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
>  mirrormaker.log:7597:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-4 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)You can see they are trying to seek to a position and thus overriding the latest offset
>  
> You can ssee it is doing a seek to position 0 for every partition. which is not what I expected



--
This message was sent by Atlassian Jira
(v8.3.4#803005)