[jira] [Commented] (KAFKA-3487) Support per-connector/per-task classloaders in Connect

["Randall Hauch (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336254#comment-15336254 ] 

Randall Hauch commented on KAFKA-3487:
--------------------------------------

I hope this can move toward an approach that can eventually support the Jigsaw module system that appears to be coming in JDK9. Jigsaw relies upon the standard Java service loader mechanism (added in JDK6) to find all implementations of an interface, and Jigsaw will then properly handle all dependencies and class loading.

A pre-JDK9 approach is to use a configuration parameter for the workers (e.g., a connector module path) that simply lists the directories in which Kafka Connect can find each connector "module". The connector module's directory would contain all of the JARs required by the connector. Upon startup, Kafka Connect could iterate through this list of paths, and for each: create a URL classloader (that inherits the parent classpath), pass the URL classloader to the service loader method to load the connector implementation class (without having to know its name), create a {{ConnectorModule}} object with the URL classloader and reference to the connector class, and then load the {{ConnectorModule}} into an internal registry keyed by name. Then, the rest of Kafka Connect would simply use the registry.

Really, this same registry could also be used to find all implementations available on the current classloader, meaning it would work with connectors that don't define a {{META-INF/services/org.apache.kafka.connect.connector.Connector}} file for the service loader.

And, to add support for JDK9, upon startup Kafka Connect would simply use the service loader to locate all {{Connector}} implementation classes, and populate the same registry, using a different {{ConnectorModule}} implementation that simply instantiates the class and relying upon JDK9 modules to properly handle all of the classloading and isolation. The rest of the Kafka Connect remains unchanged.

The only change to connector implementations is to add support for the service loader, and since that's going to be required by JDK9 it might be worth doing now. And, doing this now would greatly simply the implementation.


> Support per-connector/per-task classloaders in Connect
> ------------------------------------------------------
>
>                 Key: KAFKA-3487
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3487
>             Project: Kafka
>          Issue Type: New Feature
>          Components: KafkaConnect
>    Affects Versions: 0.10.0.0
>            Reporter: Ewen Cheslack-Postava
>            Assignee: Liquan Pei
>            Priority: Critical
>              Labels: needs-kip
>
> Currently we just use the default ClassLoader in Connect. However, this limits how we can compatibly load conflicting connector plugins. Ideally we would use a separate class loader per connector/task that is instantiated to avoid potential conflicts.
> Note that this also opens up options for other ways to provide jars to instantiate connectors. For example, Spark uses this to dynamically publish classes defined in the REPL and load them via URL: https://ardoris.wordpress.com/2014/03/30/how-spark-does-class-loading/ But much simpler examples (include URL in the connector class instead of just class name) are also possible and could be a nice way to more support dynamic sets of connectors, multiple versions of the same connector, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)