[jira] [Commented] (KAFKA-15841) Add Support for Topic-Level Partitioning in Kafka Connect

["Henrique Mota (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786731#comment-17786731 ] 

Henrique Mota commented on KAFKA-15841:
---------------------------------------

Currently, in our setup, we're unable to assign more than one worker per sink. Whenever we attempt to allocate multiple workers, all topics end up on Worker 0 because all topics have only one partition.

> Add Support for Topic-Level Partitioning in Kafka Connect
> ---------------------------------------------------------
>
>                 Key: KAFKA-15841
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15841
>             Project: Kafka
>          Issue Type: Improvement
>          Components: connect
>            Reporter: Henrique Mota
>            Priority: Trivial
>
> In our organization, we utilize JDBC sink connectors to consume data from various topics, where each topic is dedicated to a specific tenant with a single partition. Recently, we developed a custom sink based on the standard JDBC sink, enabling us to pause consumption of a topic when encountering problematic records.
> However, we face limitations within Kafka Connect, as it doesn't allow for appropriate partitioning of topics among workers. We attempted a workaround by breaking down the topics list within the 'topics' parameter. Unfortunately, Kafka Connect overrides this parameter after invoking the {{taskConfigs(int maxTasks)}} method from the {{org.apache.kafka.connect.connector.Connector}} class.
> We request the addition of support in Kafka Connect to enable the partitioning of topics among workers without requiring a fork. This enhancement would facilitate better load distribution and allow for more flexible configurations, particularly in scenarios where topics are dedicated to different tenants.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)