You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Magesh kumar Nandakumar (Jira)" <ji...@apache.org> on 2019/08/19 22:02:00 UTC
[jira] [Updated] (KAFKA-8819) Plugin path for converters not
working as intended
[ https://issues.apache.org/jira/browse/KAFKA-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Magesh kumar Nandakumar updated KAFKA-8819:
-------------------------------------------
Description:
KafakConnect allows all plugins to be available via a plugin path mechanism. This allows for classpath isolation. This is not working as designed under the following circumstances for Converters
I have 2 directories under plugin path `connector1` and `connector2`. I intend to use AvroConverter and its available in both the plugin directories. Under these circumstances, the Worker attempts to create the Converter available in the plugin director first which should ideally be deterministic but it's not because of the following reasons:-
[https://github.com/apache/kafka/blob/aa4ba8eee8e6f52a9d80a98fb2530b5bcc1b9a11/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L421] would lead to all configs of type Class to be loaded and this would mean that they are not loaded in the context of the connectors plugin loader. IIUC, the current loaded would be the DelegatingClassLoader. This would mean that the AvroConverter could potentially be loaded from connector2 plugin path while loading the class. This should be made deterministic as intended. Also, there could be instances where the Converter itself could potentially be using ServiceLoader and for that to work when the Converter is created & configured the current class loader should be set to the one corresponding to the converter.
It might also be better to load all the config in the connector config in the context of connector's plugin loader to avoid issues with classes referenced across class loader.
was:
KafakConnect allows all plugins to be available via a plugin path mechanism. This allows for classpath isolation. This is not working as designed under the following circumstances for Converters
I have 2 directories under plugin path `connector1` and `connector2`. I intend to use AvroConverter and its available in both the plugin directories. Under these circumstances, the Worker attempts to create the Converter available in the plugin director first which should ideally be deterministic but it's not because of the following reasons:-
[https://github.com/apache/kafka/blob/aa4ba8eee8e6f52a9d80a98fb2530b5bcc1b9a11/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L421] would lead to all configs of type Class to be loaded and this would mean that they are not loaded in the context of the connectors plugin loader. IIUC, the current loaded would be the DelegatingClassLoader . This would mean that the AvroConverter could potentially be loaded from connector2 plugin path while loading the class. This should be made deterministic as intended. Also, there could be instances where the Converter itself could potentially be using ServiceLoader and for that to work when the Converter is created & configured the current class loader should be set to the one corresponding to the converter.
> Plugin path for converters not working as intended
> --------------------------------------------------
>
> Key: KAFKA-8819
> URL: https://issues.apache.org/jira/browse/KAFKA-8819
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Reporter: Magesh kumar Nandakumar
> Assignee: Magesh kumar Nandakumar
> Priority: Major
>
> KafakConnect allows all plugins to be available via a plugin path mechanism. This allows for classpath isolation. This is not working as designed under the following circumstances for Converters
>
> I have 2 directories under plugin path `connector1` and `connector2`. I intend to use AvroConverter and its available in both the plugin directories. Under these circumstances, the Worker attempts to create the Converter available in the plugin director first which should ideally be deterministic but it's not because of the following reasons:-
>
> [https://github.com/apache/kafka/blob/aa4ba8eee8e6f52a9d80a98fb2530b5bcc1b9a11/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L421] would lead to all configs of type Class to be loaded and this would mean that they are not loaded in the context of the connectors plugin loader. IIUC, the current loaded would be the DelegatingClassLoader. This would mean that the AvroConverter could potentially be loaded from connector2 plugin path while loading the class. This should be made deterministic as intended. Also, there could be instances where the Converter itself could potentially be using ServiceLoader and for that to work when the Converter is created & configured the current class loader should be set to the one corresponding to the converter.
>
> It might also be better to load all the config in the connector config in the context of connector's plugin loader to avoid issues with classes referenced across class loader.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)