You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "jacob shao (JIRA)" <ji...@apache.org> on 2019/08/01 14:21:00 UTC
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898120#comment-16898120 ]
jacob shao commented on KAFKA-7500:
-----------------------------------
Hey,
I use kubernettes to spin up 2 kafka clusters locally (3 brokers each). And then I run MM2 locally as well to sync topic messages. When I send message to source topic, source.topic is created in the sink cluster, but message is not delivered. An exception is thrown in the console (see below). When I restart MM2, message arrives in the source.topic. Anyone recognize this error? Moreover, when I move one of the kafka cluster to a different machine, everything works again. I tried to increase network/io threads in the local setup, it still doesn't solve the issue.
[2019-08-01 10:18:29,033] INFO WorkerSourceTask\{id=MirrorSourceConnector-0} flushing 21 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:418)
[2019-08-01 10:18:29,072] INFO WorkerSourceTask\{id=MirrorSourceConnector-0} Finished commitOffsets successfully in 39 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:500)
[2019-08-01 10:18:29,072] ERROR WorkerSourceTask\{id=MirrorSourceConnector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179)
{color:#FF0000}java.lang.NullPointerException{color}
{color:#FF0000} at org.apache.kafka.connect.mirror.MirrorSourceTask.poll(MirrorSourceTask.java:140){color}
{color:#FF0000} at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:245){color}
{color:#FF0000} at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:221){color}
{color:#FF0000} at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177){color}
{color:#FF0000} at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227){color}
{color:#FF0000} at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515){color}
{color:#FF0000} at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264){color}
{color:#FF0000} at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128){color}
{color:#FF0000} at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628){color}
at java.base/java.lang.Thread.run(Thread.java:834)
[2019-08-01 10:18:29,073] ERROR WorkerSourceTask\{id=MirrorSourceConnector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:180)
[2019-08-01 10:18:29,073] INFO [Producer clientId=connector-producer-MirrorSourceConnector-0] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer:1153)
[2019-08-01 10:18:29,080] INFO [Producer clientId=producer-7] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer:1153)
[2019-08-01 10:18:44,602] INFO [Worker clientId=connect-2, groupId=sec-mm2] Attempt to heartbeat failed since coordinator localhost:31000 (id: 2147483647 rack: null) is either not started or not valid. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:931)
[2019-08-01 10:18:44,602] INFO [Worker clientId=connect-2, groupId=sec-mm2] Group coordinator localhost:31000 (id: 2147483647 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:780)
[2019-08-01 10:18:44,612] INFO [Worker clientId=connect-2, groupId=sec-mm2] Discovered group coordinator localhost:31002 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:728)
> MirrorMaker 2.0 (KIP-382)
> -------------------------
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
> Reporter: Ryanne Dolan
> Priority: Minor
> Fix For: 2.4.0
>
> Attachments: Active-Active XDCR setup.png
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)