[jira] [Created] (KAFKA-10582) Mirror Maker 2 not replicating new topics until restart

["Robert Martin (Jira)" <ji...@apache.org>]

Robert Martin created KAFKA-10582:
-------------------------------------

             Summary: Mirror Maker 2 not replicating new topics until restart
                 Key: KAFKA-10582
                 URL: https://issues.apache.org/jira/browse/KAFKA-10582
             Project: Kafka
          Issue Type: Bug
          Components: mirrormaker
    Affects Versions: 2.5.1
         Environment: RHEL 7 Linux.
            Reporter: Robert Martin


We are using Mirror Maker 2 from the 2.5.1 release for replication on some clusters.  Replication is working as expected for existing topics.  When we create a new topic, however, Mirror Maker 2 creates the replicated topic as expected but never starts replicating it.  If we restart Mirror Maker 2 within 2-3 minutes the topic starts replicating as expected.  From documentation we haveve seen it appears this should start replicating without a restart based on the settings we have.

*Example:*
Create topic "mytesttopic" on source cluster
MirrorMaker 2 creates "source.mytesttopioc" on target cluster with no issue
MirrorMaker 2 does not replicate "mytesttopic" -> "source.mytesttopic"
Restart MirrorMaker 2 and now replication works for "mytesttopic" -> "source.mytesttopic"

*Example config:*
name = source->target
group.id = source-to-target

# specify any number of cluster aliases
clusters = source, target

# connection information for each cluster
# This is a comma separated host:port pairs for each cluster
# for e.g. "A_host1:9092, A_host2:9092, A_host3:9092"
source.bootstrap.servers = sourcehosts:9092
target.bootstrap.servers = targethosts:9092

# enable and configure individual replication flows
source->target.enabled = true

# regex which defines which topics gets replicated. For eg "foo-.*"
source->target.topics = .*

target->source = false
target->source.topics = .*

# Setting replication factor of newly created remote topics
replication.factor=3

############################# Internal Topic Settings  #############################
# The replication factor for mm2 internal topics "heartbeats", "B.checkpoints.internal" and
# "mm2-offset-syncs.B.internal"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
checkpoints.topic.replication.factor=3
heartbeats.topic.replication.factor=3
offset-syncs.topic.replication.factor=3

# The replication factor for connect internal topics "mm2-configs.B.internal", "mm2-offsets.B.internal" and
# "mm2-status.B.internal"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offset.storage.replication.factor=3
status.storage.replication.factor=3
config.storage.replication.factor=3

# customize as needed
# replication.policy.separator = _
# sync.topic.acls.enabled = false
# emit.heartbeats.interval.seconds = 5

# tasks.max results in parallelism
tasks.max = 16
refresh.topics.enabled = true
sync.topic.configs.enabled = true
# Setting the below too low can result in performance issues
refresh.topics.interval.seconds = 300
refresh.groups.interval.seconds = 300
readahead.queue.capacity = 100

emit.checkpoints.enabled = true
emit.checkpoints.interval.seconds = 5



--
This message was sent by Atlassian Jira
(v8.3.4#803005)