[jira] [Commented] (KAFKA-18211) ClassGraph scanning does not correctly find isolated connect plugins

["Snehashis Pal (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-18211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17910357#comment-17910357 ] 

Snehashis Pal commented on KAFKA-18211:
---------------------------------------

Raised [KAFKA-18211: Override class loaders for class graph scanning in connect. by snehashisp · Pull Request #18403 · apache/kafka|https://github.com/apache/kafka/pull/18403], [~gharris1727] Please approve the test run.

> ClassGraph scanning does not correctly find isolated connect plugins
> --------------------------------------------------------------------
>
>                 Key: KAFKA-18211
>                 URL: https://issues.apache.org/jira/browse/KAFKA-18211
>             Project: Kafka
>          Issue Type: Bug
>          Components: connect
>    Affects Versions: 4.0.0
>            Reporter: Snehashis Pal
>            Priority: Blocker
>             Fix For: 4.0.0
>
>
> Connect used to use reflections scanner for scanning and identifying connect plugins in its plugin.path. This would load said plugins in isolation via the use of a child first PluginClassloader, which is designed to load class from its set of URIs before delegating to parent, if not found. This effectively enforces that if a plugin and its dependencies are part of a plugin path it would not conflict with other plugins in the plugin path or plugins in classpath. 
>  
> GlassGraph was introduced as a replacement for the older reflections scanner in [KAFKA-15203 Use Classgraph since org.reflections is no longer under maintainence by PARADOXST · Pull Request #16604 · apache/kafka|https://github.com/apache/kafka/pull/16604]. It is used in place of reflections scanner for finding plugins during plugin scanning. The issue here is that it is missing any plugins present in isolated plugin paths if its already present in classpath.  We can repro this by adding the json converter under an isolated plugin path and starting connect with debug logs. We can see the logs from ReflectionsScanner and find that the ClassGraph loader is always fetching the plugin from the classpath even though the PluginClassLoader is provided. This is causing [kafka/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/ReflectionScanner.java at 520681c38dbefe497181c4fd5dfc793d54233408 · apache/kafka|https://github.com/apache/kafka/blob/520681c38dbefe497181c4fd5dfc793d54233408/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/ReflectionScanner.java#L134] check to fail with the logs like. 
> {code:java}
> [2024-12-11 07:29:28,968] DEBUG class org.apache.kafka.connect.json.JsonConverter from other classloader jdk.internal.loader.ClassLoaders$AppClassLoader@c387f44 is visible from C:\Users\user\Desktop\confluent\testing\plugins3\connect-file-1.2.1-T-0.9.0-P-3.1, excluding to prevent isolated loading (org.apache.kafka.connect.runtime.isolation.ReflectionScanner:135)
> [2024-12-11 07:29:28,969] DEBUG class org.apache.kafka.connect.json.JsonConverter from other classloader jdk.internal.loader.ClassLoaders$AppClassLoader@c387f44 is visible from C:\Users\user\Desktop\confluent\testing\plugins3\connect-file-1.2.1-T-0.9.0-P-3.1, excluding to prevent isolated loading (org.apache.kafka.connect.runtime.isolation.ReflectionScanner:135) {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)