You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@kafka.apache.org by "Chris Egerton (Jira)" <ji...@apache.org> on 2022/11/15 21:28:00 UTC

[jira] [Resolved] (KAFKA-13764) Potential improvements for Connect incremental rebalancing logic

     [ https://issues.apache.org/jira/browse/KAFKA-13764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Egerton resolved KAFKA-13764.
-----------------------------------
    Resolution: Fixed

> Potential improvements for Connect incremental rebalancing logic
> ----------------------------------------------------------------
>
>                 Key: KAFKA-13764
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13764
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Chris Egerton
>            Assignee: Chris Egerton
>            Priority: Minor
>
> There are a few small changes that we might make to the incremental rebalancing logic for Kafka Connect to improve distribution of connectors and tasks across a cluster and address potential bugs:
>  # During assignment, assign new connectors and tasks across the cluster before calculating revocations that may be necessary in order to balance the cluster. This way, we can potentially skip a round of revocation by using newly-created connectors and tasks to balance out the cluster.
>  # Perform connector and task revocation in more cases, such as when one or more connectors are reconfigured to use fewer tasks, which can possibly lead to an imbalanced cluster.
>  # Fix [this line|https://github.com/apache/kafka/blob/06ca4850c5b2b12e972f48e03fe4f9c1032f9a3e/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/IncrementalCooperativeAssignor.java#L248] to use the same aggregation logic that's used [here|https://github.com/apache/kafka/blob/06ca4850c5b2b12e972f48e03fe4f9c1032f9a3e/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/IncrementalCooperativeAssignor.java#L273-L281] in order to avoid overwriting map values when they should be combined instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)