[jira] [Commented] (KAFKA-6208) Reduce startup time for Kafka Connect workers

["Satyajit varma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-6208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357745#comment-16357745 ] 

Satyajit varma commented on KAFKA-6208:
---------------------------------------

[~rhauch] , we could make the connectors,SMT's and converters load asynchronously, like below,

{code}

CompletableFuture.runAsync(()->{
 addPlugins(plugins.connectors(), loader);
 connectors.addAll(plugins.connectors());
}, Executors.newSingleThreadExecutor());


CompletableFuture.runAsync(()->{
 addPlugins(plugins.converters(), loader);
 converters.addAll(plugins.converters());
}, Executors.newSingleThreadExecutor());


CompletableFuture.runAsync(()->{
 addPlugins(plugins.transformations(), loader);
 transformations.addAll(plugins.transformations());
}, Executors.newSingleThreadExecutor());

{code}

> Reduce startup time for Kafka Connect workers
> ---------------------------------------------
>
>                 Key: KAFKA-6208
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6208
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 1.0.0
>            Reporter: Randall Hauch
>            Priority: Major
>
> Kafka Connect startup times are excessive with a handful of connectors on the plugin path or classpath. We should not be scanning three times (once for connectors, once for SMTs, and once for converters), and hopefully we can avoid scanning directories that are clearly not plugin directories. 
> We should also consider using Java's Service Loader to quickly identify connectors. The latter would require a KIP and would require time to for connectors to migrate, but we could be smarter about only scanning plugin directories that need to be scanned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)