You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Chris Egerton (Jira)" <ji...@apache.org> on 2024/08/03 00:28:00 UTC
[jira] [Created] (KAFKA-17252) Forwarded source task zombie fencings may fail when leader has just started
Chris Egerton created KAFKA-17252:
-------------------------------------
Summary: Forwarded source task zombie fencings may fail when leader has just started
Key: KAFKA-17252
URL: https://issues.apache.org/jira/browse/KAFKA-17252
Project: Kafka
Issue Type: Bug
Components: connect
Reporter: Chris Egerton
Assignee: Chris Egerton
We've observed some flaky integration test failures such as [this one|https://ge.apache.org/s/52il7msnknzp2/tests/task/:connect:mirror:test/details/org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest/testMultiNodeCluster()?top-execution=1] where a source task fails to start with exactly-once support enabled with this stack trace:
{code:java}
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: This worker is still starting up and has not been able to read a session key from the config topic yet
at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:186)
at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:140)
at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:101)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$fenceZombieSourceTasks$23(DistributedHerder.java:1329)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583) {code}
This occurs because the leader has not yet read (or believes it has not yet read) a session key from the config topic.
However, in a cluster where all nodes have always used the sessioned rebalance protocol, this scenario should be impossible: there must be a session key present in the topic in order for a leader to handle external requests (such as creating connectors), and all workers must read to the end of all internal topics before joining the cluster.
The cause of this failure is that, during startup, session keys read from the config topic are ignored. The herder does [check its config state snapshot for a session key|https://github.com/apache/kafka/blob/da14b5a61dc90fc70748278c98ce312a7a433c0d/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L463] in its tick thread loop, which should help, but it's possible that a worker joins the cluster, becomes the leader, and receives a request from a follower to fence a zombie source task before this check occurs, which will then cause the leader to response with a 503 error, failing the task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)