Regarding Kafka connect task to partition relationship for both source and sink connectors

[Yeikel Santana <em...@yeikel.com>]

Hi everyone,


From my understanding, if a topic has  n partitions, we can create up to n tasks for both the source and sink connectors to achieve the maximum parallelism. Adding more tasks would not be beneficial, as they would remain idle and be limited to the number of partitions of the topic


Could you please confirm if this understanding is correct?


If this understanding is incorrect could you please explain the relationship if any?


Thank you!

Re: Regarding Kafka connect task to partition relationship for both source and sink connectors

[Alex Craig <al...@gmail.com>]

For sink connectors, I believe you can scale up the tasks to match the
partitions on the topic.  But I don't believe this is the case for source
connectors; the number of partitions on the topic you're producing to has
nothing to do with the number of connector tasks.  It really depends on the
individual source connector and if the source data-type could benefit from
multiple tasks.  For example, the JDBC source connector (a very popular
connector) only supports 1 task - even if you're querying multiple tables.

Bottom line: you'll need to check the documentation for the connector in
question to see if it supports multiple tasks.

Alex

On Thu, May 30, 2024 at 7:51 AM Sébastien Rebecchi <sr...@kameleoon.com>
wrote:

> Hello
>
> Confirmed. Partition is the minimal granularity level, so having more
> consumers than the number of partitions of a topic for a same consumer
> group is useless, having P partitions means maximum parallelism is reached
> using P consumers.
>
> Regards,
>
> Sébastien.
>
> Le jeu. 30 mai 2024 à 14:43, Yeikel Santana <em...@yeikel.com> a écrit :
>
> > Hi everyone,
> >
> >
> > From my understanding, if a topic has  n partitions, we can create up to
> n
> > tasks for both the source and sink connectors to achieve the maximum
> > parallelism. Adding more tasks would not be beneficial, as they would
> > remain idle and be limited to the number of partitions of the topic
> >
> >
> > Could you please confirm if this understanding is correct?
> >
> >
> > If this understanding is incorrect could you please explain the
> > relationship if any?
> >
> >
> > Thank you!
> >
> >
> >
>

Re: Regarding Kafka connect task to partition relationship for both source and sink connectors

[Sébastien Rebecchi <sr...@kameleoon.com>]

Hello

Confirmed. Partition is the minimal granularity level, so having more
consumers than the number of partitions of a topic for a same consumer
group is useless, having P partitions means maximum parallelism is reached
using P consumers.

Regards,

Sébastien.

Le jeu. 30 mai 2024 à 14:43, Yeikel Santana <em...@yeikel.com> a écrit :

> Hi everyone,
>
>
> From my understanding, if a topic has  n partitions, we can create up to n
> tasks for both the source and sink connectors to achieve the maximum
> parallelism. Adding more tasks would not be beneficial, as they would
> remain idle and be limited to the number of partitions of the topic
>
>
> Could you please confirm if this understanding is correct?
>
>
> If this understanding is incorrect could you please explain the
> relationship if any?
>
>
> Thank you!
>
>
>

Re: [EXTERNAL] Regarding Kafka connect task to partition relationship for both source and sink connectors

["Tauzell, Dave" <Da...@surescripts.com>]

The docs say:  “Each task is assigned to a thread. Each task is capable of handling multiple Kafka partitions, but a single partition must be handled by only one task.”    From what I understand additional tasks would sit idle.



From: Yeikel Santana <em...@yeikel.com>
Date: Thursday, May 30, 2024 at 7:43 AM
To: users@kafka.apache.org <us...@kafka.apache.org>
Subject: [EXTERNAL] Regarding Kafka connect task to partition relationship for both source and sink connectors
Hi everyone,


From my understanding, if a topic has  n partitions, we can create up to n tasks for both the source and sink connectors to achieve the maximum parallelism. Adding more tasks would not be beneficial, as they would remain idle and be limited to the number of partitions of the topic


Could you please confirm if this understanding is correct?


If this understanding is incorrect could you please explain the relationship if any?


Thank you!

This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.