[jira] [Created] (KAFKA-6467) Enforce layout of dependencies within a Connect plugin to be deterministic

["Konstantine Karantasis (JIRA)" <ji...@apache.org>]

Konstantine Karantasis created KAFKA-6467:
---------------------------------------------

             Summary: Enforce layout of dependencies within a Connect plugin to be deterministic
                 Key: KAFKA-6467
                 URL: https://issues.apache.org/jira/browse/KAFKA-6467
             Project: Kafka
          Issue Type: Bug
          Components: KafkaConnect
    Affects Versions: 0.11.0.2, 1.0.0
            Reporter: Konstantine Karantasis
            Assignee: Konstantine Karantasis
             Fix For: 1.0.1, 0.11.0.3


In principle, Connect plugins that intend to load their dependencies in isolation should not contain any conflicts among the classes they package as dependencies. In other words, the order in which a plugin's dependencies are laid out and passed to its plugin class loader should not matter.

However, in practice, there are rare and suboptimal situations where a plugin needs to bundle a few packages with conflicting dependencies because it doesn't control packaging of third-party modules. In those cases depending on a deterministic ordering within the class loader's path can help the Connect plugin enforce loading of the desired classes as needed. (For example, see [HDFS connector with MapR libs|https://github.com/confluentinc/kafka-connect-hdfs/issues/270] or [HDFS connector with Hive's extended jar|https://github.com/confluentinc/kafka-connect-hdfs/issues/261)

To achieve such ordering, this improvement suggests ordering a plugin's dependencies in a nested directory structure by sorting such paths alphanumerically. This way the deterministic order is implicit (no extra configuration is required) and a specific dependency can be put earlier or later in the class loader's path with appropriate naming of its package path (e.g. within the plugin's directory).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)