[jira] [Created] (KAFKA-14072) Crashed MirrorCheckpointConnector appears as running in REST API

["Mickael Maison (Jira)" <ji...@apache.org>]

Mickael Maison created KAFKA-14072:
--------------------------------------

             Summary: Crashed MirrorCheckpointConnector appears as running in REST API
                 Key: KAFKA-14072
                 URL: https://issues.apache.org/jira/browse/KAFKA-14072
             Project: Kafka
          Issue Type: Bug
          Components: KafkaConnect, mirrormaker
            Reporter: Mickael Maison


In one cluster I had a partially crashed MirrorCheckpointConnector instance. It had stopped mirroring offsets and emitting metrics completely but the connector and its single task were still reporting as running in the REST API.

Looking at the logs, I found this stacktrace:

{code:java}
java.lang.NullPointerException
	at org.apache.kafka.connect.mirror.MirrorCheckpointTask.checkpoint(MirrorCheckpointTask.java:187)
	at org.apache.kafka.connect.mirror.MirrorCheckpointTask.lambda$checkpointsForGroup$2(MirrorCheckpointTask.java:171)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
	at java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1764)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at org.apache.kafka.connect.mirror.MirrorCheckpointTask.checkpointsForGroup(MirrorCheckpointTask.java:173)
	at org.apache.kafka.connect.mirror.MirrorCheckpointTask.sourceRecordsForGroup(MirrorCheckpointTask.java:157)
	at org.apache.kafka.connect.mirror.MirrorCheckpointTask.poll(MirrorCheckpointTask.java:139)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:291)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:248)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:186)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:241)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
WARN [prod-source->sc-prod-target.MirrorCheckpointConnector|task-0] Failure polling consumer state for checkpoints. (org.apache.kafka.connect.mirror.MirrorCheckpointTask) [task-thread-prod-source->sc-prod-target.MirrorCheckpointConnector-0]
{code}

Not sure if it's related but prior this exception, there's quite a lot of:

{code:java}
ERROR [prod-source->sc-prod-target.MirrorCheckpointConnector|task-0] WorkerSourceTask{id=prod-source->sc-prod-target.MirrorCheckpointConnector-0} failed to send record to prod-source.checkpoints.internal:  (org.apache.kafka.connect.runtime.WorkerSourceTask) [kafka-producer-network-thread | connector-producer-prod-source->sc-prod-target.MirrorCheckpointConnector-0]
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
	at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:760)
	at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:747)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
	at java.base/java.lang.Thread.run(Thread.java:829)
{code}

and some users had started consumers in the target cluster hence causing these log lines:

{code:java}
ERROR [prod-source->sc-prod-target.MirrorCheckpointConnector|task-0] [AdminClient clientId=adminclient-137] OffsetCommit request for group id <GROUP_ID> and partition <TP> failed due to unexpected error UNKNOWN_MEMBER_ID. (org.apache.kafka.clients.admin.internals.AlterConsumerGroupOffsetsHandler) [kafka-admin-client-thread | adminclient-137]
{code}

Unfortunately I don't have the full history, so it's unclear if this happened while stopping but the connector stayed in this state for several hours until it was explicitly deleted via the REST API.






--
This message was sent by Atlassian Jira
(v8.20.10#820010)