You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "George Yang (Jira)" <ji...@apache.org> on 2025/01/01 08:56:00 UTC
[jira] [Created] (KAFKA-18386) Mirror Maker2 Pod CrashLoopBackoff When one DC is powered off
George Yang created KAFKA-18386:
-----------------------------------
Summary: Mirror Maker2 Pod CrashLoopBackoff When one DC is powered off
Key: KAFKA-18386
URL: https://issues.apache.org/jira/browse/KAFKA-18386
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 3.7.1
Reporter: George Yang
When using Kubernetes deployment with MirrorMaker v3.7.1 and deploying one Kafka node in each data center (DC1 and DC2), if DC1 is powered off, DC2 will encounter a CrashLoopBackOff error. This issue is different from the one described in KAFKA-17784. Please find the report log below:
```log
[2025-01-01 08:05:53,432] WARN [AdminClient clientId=dc64->dc88] Connection to node -1 (/192.168.2.88:13399) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient:830)[kafka-admin-client-thread | dc64->dc88]
[2025-01-01 08:05:55,652] INFO [AdminClient clientId=dc64->dc88] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:267)[kafka-admin-client-thread | dc64->dc88]
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
[2025-01-01 08:05:55,653] INFO App info kafka.admin.client for dc64->dc88 unregistered (org.apache.kafka.common.utils.AppInfoParser:88)[kafka-admin-client-thread | dc64->dc88]
[2025-01-01 08:05:55,653] INFO [AdminClient clientId=dc64->dc88] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:267)[kafka-admin-client-thread | dc64->dc88]
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
[2025-01-01 08:05:55,653] INFO [AdminClient clientId=dc64->dc88] Timed out 1 remaining operation(s) during close. (org.apache.kafka.clients.admin.KafkaAdminClient:1450)[kafka-admin-client-thread | dc64->dc88]
[2025-01-01 08:05:55,657] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics:684)[kafka-admin-client-thread | dc64->dc88]
[2025-01-01 08:05:55,658] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics:688)[kafka-admin-client-thread | dc64->dc88]
[2025-01-01 08:05:55,658] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics:694)[kafka-admin-client-thread | dc64->dc88]
[2025-01-01 08:05:55,658] ERROR Stopping due to error (org.apache.kafka.connect.mirror.MirrorMaker:360)[main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:305)
at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:285)
at org.apache.kafka.connect.runtime.WorkerConfig.kafkaClusterId(WorkerConfig.java:415)
at org.apache.kafka.connect.mirror.MirrorMaker.addHerder(MirrorMaker.java:252)
at java.base/java.lang.Iterable.forEach(Unknown Source)
at org.apache.kafka.connect.mirror.MirrorMaker.<init>(MirrorMaker.java:158)
at org.apache.kafka.connect.mirror.MirrorMaker.<init>(MirrorMaker.java:170)
at org.apache.kafka.connect.mirror.MirrorMaker.<init>(MirrorMaker.java:174)
at org.apache.kafka.connect.mirror.MirrorMaker.main(MirrorMaker.java:347)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:299)
... 8 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
[2025-01-01 08:05:55,687] INFO Stopped http_8083@6705fb02\{HTTP/1.1, (http/1.1)}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:383)[JettyShutdownThread]
```
The configuration of mirrormaker is:
```
clusters = dc64, dc88
dc64.bootstrap.servers = 192.168.2.64:13399
dc88.bootstrap.servers = 192.168.2.88:13399
dc64->dc88.enabled = true
dc64->dc88.topics = .*
dc88->dc64.enabled = true
dc88->dc64.topics = .*
replication.factor=1
tasks.max=6
emit.checkpoints.interval.seconds=5
dc64.producer.acks=all
dc64.producer.batch.size=50000
dc64.consumer.auto.offset.reset=latest
dc88.consumer.auto.offset.reset=latest
dc64.consumer.max.poll.interval.ms=20000
dc88.consumer.max.poll.interval.ms=20000
refresh.topics.enabled=true
refresh.topics.interval.seconds=5
refresh.groups.enabled=true
refresh.groups.interval.seconds=5
dedicated.mode.enable.internal.rest = true
dc64.scheduled.rebalance.max.delay.ms=20000
dc88.scheduled.rebalance.max.delay.ms=20000
checkpoints.topic.replication.factor=1
heartbeats.topic.replication.factor=1
offset-syncs.topic.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
config.storage.replication.factor=1
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)