You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@kafka.apache.org by "Chris Egerton (Jira)" <ji...@apache.org> on 2023/06/07 14:26:00 UTC

[jira] [Updated] (KAFKA-15059) Exactly-once source tasks fail to start during pending rebalances

     [ https://issues.apache.org/jira/browse/KAFKA-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Egerton updated KAFKA-15059:
----------------------------------
    Affects Version/s: 3.6.0
                           (was: 3.3.0)
                           (was: 3.4.0)
                           (was: 3.3.1)
                           (was: 3.3.2)
                           (was: 3.5.0)
                           (was: 3.4.1)

> Exactly-once source tasks fail to start during pending rebalances
> -----------------------------------------------------------------
>
>                 Key: KAFKA-15059
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15059
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect, mirrormaker
>    Affects Versions: 3.6.0
>            Reporter: Chris Egerton
>            Assignee: Chris Egerton
>            Priority: Major
>
> When asked to perform a round of zombie fencing, the distributed herder will [reject the request|https://github.com/apache/kafka/blob/17fd30e6b457f097f6a524b516eca1a6a74a9144/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1249-L1250] if a rebalance is pending, which can happen if (among other things) a config for a new connector or a new set of task configs has been recently read from the config topic.
> Normally this can be alleviated with a simple task restart, which isn't great but isn't terrible.
> However, when running MirrorMaker 2 in dedicated mode, there is no API to restart failed tasks, and it can be more common to see this kind of failure on a fresh cluster because three connector configurations are written in rapid succession to the config topic.
>  
> In order to provide a better experience for users of both vanilla Kafka Connect and dedicated MirrorMaker 2 clusters, we can retry (likely with the same exponential backoff introduced with KAFKA-14732) zombie fencing attempts that fail due to a pending rebalance.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)