[jira] [Commented] (KAFKA-3848) Mirror Maker sometimes fails to mirror first batch of messages

["James Cheng (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332480#comment-15332480 ] 

James Cheng commented on KAFKA-3848:
------------------------------------

This behavior may be related to the value of auto.offset.reset. Mirrormaker uses the default value of auto.offset.reset, which is "largest" (aka, start receiving the newest messages in the stream)

I suspect what is happening is this:
1) mirrormaker has a wildcard listening to dc1
2) First message gets sent to dc1, topic gets auto-created
3) mirrormaker gets notified that a new topic is created (because of the wildcard)
4) mirrormaker starts consuming from the topic at "largest". It connects to the end of the topic and waits for new messages to come in.
5) second message gets sent to dc1
6) mirrormaker sees the 2nd message.

You said that when you consume from the topic in dc1, you see both messages. How were you listening to that topic? I suspect that if you consume from dc1 using a wildcarded consumer, that you might see similar behavior to what you are seeing from mirrormaker. Your wildcarded consumer would see just the 2nd message.


> Mirror Maker sometimes fails to mirror first batch of messages
> --------------------------------------------------------------
>
>                 Key: KAFKA-3848
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3848
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.9.0.1, 0.10.0.0
>            Reporter: James Clarke
>
> I am seeing an intermittent issue in Mirror Maker where the first batch of messages are not always mirrored to the target cluster. All messages after the first batch are mirrored.
> I have a github repo ([jc/kafka-mirror-maker-test|https://github.com/jc/kafka-mirror-maker-test]) which reproduces the issue using Confluent's docker containers (running 0.10.0). However on our environment we are using our own kafka containers running 0.9.0.1.
> Environment:
> - edge datacenter dc1. 1 zk server, 1 kafka server.
> - aggregate datacenter dc2. 1 zk server, 1 kafka server, 1 mirror maker.
> - kafka server setup to auto create topics
> - Mirrror maker configured mirror from dc1 to dc2 using a whitelist containing both explicitly topics and regex topics.
> Steps to reproduce:
> - Send message to a non-existent topic in dc1.
> - Send a second message to topic in dc1.
> Observed:
> - After first message the topic is not created in dc2.
> - After second message topic is present in dc2.
> - Consuming from topic in dc1 shows both messages.
> - Consuming from topic in dc2 shows only the second message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)