You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Jozef Vilcek (Jira)" <ji...@apache.org> on 2021/08/12 14:39:00 UTC
[jira] [Updated] (KAFKA-13196) MirrorMaker 2 not always starting
tasks
[ https://issues.apache.org/jira/browse/KAFKA-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jozef Vilcek updated KAFKA-13196:
---------------------------------
Summary: MirrorMaker 2 not always starting tasks (was: MirrorMaker 2 not always starting tasks after upgrade 2.4.0 -> 2.7.1)
> MirrorMaker 2 not always starting tasks
> ---------------------------------------
>
> Key: KAFKA-13196
> URL: https://issues.apache.org/jira/browse/KAFKA-13196
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.7.1
> Reporter: Jozef Vilcek
> Priority: Major
>
> I am using MirrorMaker 2.0 and running it via [ MirrorMaker.java|https://github.com/apache/kafka/blob/2.7/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java] class. This method will start up `DistributedHerder` and will use non-functional `advertisedUrl`, and therefore workers can not talk to each other and coordinate.
> After upgrading from version `2.4.0` to `2.7.1` I did notice that when I am starting up mirror maker ti does not always start tasks - just connector is executing. Doing some amount of stop and start it will eventually start tasks too.
> After a bit of digging I did noticed that in attempt to configure connector's task, code ends up in this [1] branch, where configure request is being forwarded to the leader. From some reason, task configuration is not done on leader. However, MirrorMaker does not pack RestServer and therefore that request will never succeed.
> I am not sure what is going no or why it does seem to work better on 2.4.0. I noticed that connector start procedure did involve less callback on 2.4.0 in connector start sequence [2].
> I believe the root cause is change in startup procedure of connectors in general. In vestion 2.4, when connector is started by the leader here [1] it will immediately setup and submit configuration for connector's tasks. However, in 2.7.1, is it more asynchronous. Connector is started here [2]
> Does this look like a bug? Or am I doing something wrong?
>
> [1] https://github.com/apache/kafka/blob/2.7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1494
> [2] https://github.com/apache/kafka/blob/2.4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1236
--
This message was sent by Atlassian Jira
(v8.3.4#803005)