Kafka connect distributed mode not distributing the work

[Manjunath Shetty H <ma...@live.com>]

Hi all,


I am running kafka connect using 2 node cluster. I have 5 connectors running with maxtask 1 each. But all the tasks are running in same node, work is not distributed across 2 nodes.


I am using a custom source connectors.


Any help is appreciated


Thanks

Manjunath

Re: Kafka connect distributed mode not distributing the work

[Ewen Cheslack-Postava <ew...@confluent.io>]

Hi Manjunath,

I think you're seeing a case of this issue: https://issues.apache.
org/jira/browse/KAFKA-4553 where the way round robin assignment works with
an even # of workers and connectors that generate only 1 task generates
uneven work assignments because connectors aren't really equivalent to
tasks in practice (all the connectors are getting assigned to 1 worker, all
the tasks to the other).

I've filed a PR to address the issue, though it probably won't be available
until the next release (but it'll work automatically once you upgrade). The
easiest short term solution is to simply increase to 3 worker processes
since that'll avoid assigning all the connectors to the first worker and
all tasks to the second.

-Ewen

On Wed, Dec 14, 2016 at 8:59 PM, Manjunath Shetty H <
manjunathshetty@live.com> wrote:

> Hi all,
>
>
> I am running kafka connect using 2 node cluster. I have 5 connectors
> running with maxtask 1 each. But all the tasks are running in same node,
> work is not distributed across 2 nodes.
>
>
> I am using a custom source connectors.
>
>
> Any help is appreciated
>
>
> Thanks
>
> Manjunath
>