Kafka Connect sink connector in distributed mode: how are records distributed to workers?

[Behrang Saeedzadeh <be...@gmail.com>]

Hi,

How does Kafka Connect distribute records between workers for a sink
connector when the connector is only configured to get data from one topic?

* Does it ensure all records in a given partition are sent to the same
worker instance?
* When a new worker is added to the cluster, what steps are taken by
Connect? In particular, if Connect unassigns some partitions from the given
workers and assigns them to this new worker, would it wait for all
in-flight records belonging to these partitions to be processed by the
existing workers?

Best regards,
Behrang Saeedzadeh

Re: Kafka Connect sink connector in distributed mode: how are records distributed to workers?

[Randall Hauch <rh...@gmail.com>]

Connect uses Kafka's group coordination mechanism and normal consumers to
balance/assign to the various sink tasks the topic partitions for the
subscribed topics. A task will continue to receive the records from its
assigned topic partitions until a rebalance occurs, which can happen when
any connector config is added/changed/removed or when a worker adds/leaves
the cluster.

On Sun, Sep 10, 2017 at 10:40 PM, Behrang Saeedzadeh <be...@gmail.com>
wrote:

> Hi,
>
> How does Kafka Connect distribute records between workers for a sink
> connector when the connector is only configured to get data from one topic?
>
> * Does it ensure all records in a given partition are sent to the same
> worker instance?
> * When a new worker is added to the cluster, what steps are taken by
> Connect? In particular, if Connect unassigns some partitions from the given
> workers and assigns them to this new worker, would it wait for all
> in-flight records belonging to these partitions to be processed by the
> existing workers?
>
> Best regards,
> Behrang Saeedzadeh
>