[jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change

["Ewen Cheslack-Postava (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095747#comment-16095747 ] 

Ewen Cheslack-Postava commented on KAFKA-5505:
----------------------------------------------

There's been some discussion about more incremental rebalancing, but as you add/remove tasks, there's no way to avoid the fact that to keep the work balanced we may need to stop/start/move some tasks.

Can you explain why you have task sets changing so frequently? It's possible that a different way of assigning partitions to tasks might avoid rebalancing all the time.

> Connect: Do not restart connector and existing tasks on task-set change
> -----------------------------------------------------------------------
>
>                 Key: KAFKA-5505
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5505
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 0.10.2.1
>            Reporter: Per Steffensen
>
> I am writing a connector with a frequently changing task-set. It is really not working very well, because the connector and all existing tasks are restarted when the set of tasks changes. E.g. if the connector is running with 10 tasks, and an additional task is needed, the connector itself and all 10 existing tasks are restarted, just to make the 11th task run also. My tasks have a fairly heavy initialization, making it extra annoying. I would like to see a change, introducing a "mode", where only new/deleted tasks are started/stopped when notifying the system that the set of tasks changed (calling context.requestTaskReconfiguration() - or something similar).
> Discussed this issue a little on dev@kafka.apache.org in the thread "Kafka Connect: To much restarting with a SourceConnector with dynamic set of tasks"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)