[jira] [Created] (KAFKA-13196) MirrorMaker 2 not always starting tasks after upgrade 2.4.0 -> 2.7.1

["Jozef Vilcek (Jira)" <ji...@apache.org>]

Jozef Vilcek created KAFKA-13196:
------------------------------------

             Summary: MirrorMaker 2 not always starting tasks after upgrade 2.4.0 -> 2.7.1
                 Key: KAFKA-13196
                 URL: https://issues.apache.org/jira/browse/KAFKA-13196
             Project: Kafka
          Issue Type: Bug
          Components: mirrormaker
    Affects Versions: 2.7.1
            Reporter: Jozef Vilcek


I am using MirrorMaker 2.0 and running it via [ MirrorMaker.java|https://github.com/apache/kafka/blob/2.7/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java] class. This method will start up `DistributedHerder` and will use non-functional `advertisedUrl`, and therefore workers can not talk to each other and coordinate.

After upgrading from version `2.4.0` to `2.7.1` I did notice that when I am starting up mirror maker ti does not always start tasks - just connector is executing. Doing some amount of stop and start it will eventually start tasks too.

After a bit of digging I did noticed that in attempt to configure connector's task, code ends up in this [1] branch, where configure request is being forwarded to the leader. From some reason, task configuration is not done on leader. However, MirrorMaker does not pack RestServer and therefore that request will never succeed.

I am not sure what is going no or why it does seem to work better on 2.4.0. I noticed that connector start procedure did involve less callback on 2.4.0 in connector start sequence [2].

I believe the root cause is change in startup procedure of connectors in general. In vestion 2.4, when connector is started by the leader here [1] it will immediately setup and submit configuration for connector's tasks. However, in 2.7.1, is it more asynchronous. Connector is started here [2] 

Does this look like a bug? Or am I doing something wrong?

 

[1] https://github.com/apache/kafka/blob/2.7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1494

[2] https://github.com/apache/kafka/blob/2.4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1236



--
This message was sent by Atlassian Jira
(v8.3.4#803005)