[jira] [Commented] (OOZIE-2714) Detect conflicting resources during class loading

["Peter Bacsko (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185889#comment-16185889 ] 

Peter Bacsko commented on OOZIE-2714:
-------------------------------------

I've worked on this a little bit recently.

I ran into a bunch of tricky problems. But this POC seems to do the job. It did find the conflicts when I inserted a near-duplication of a Guava class into the sharelib.

I don't know if it's possible to do this simpler. I tried to mess around with overriding only {{findClass()}} but that didn't work. Looks like it's necessary to invoke {{defineClass()}} which enforces that all dependencies of the initially loaded class (in our case, {{LauncherAM}}) will subsequently be loaded by the custom classloader, not the default AppClassLoader. Anyway, it was fun :) BTW it was also necessary to add a new main class which loads {{LauncherAM}} with reflection.

There is a second approach - use {{-Djava.system.class.loader}} in the command line but that seems to be more problematic. It's because we have to check {{java.class.path}} and construct the {{URL[]}} array by hand. To do this, we either re-use code from {{sun.misc.*}} packages or copy-paste some code from there to do it the same way. Neither seems to be a convincing solution, so I'd vote for the reflection-based approach.

> Detect conflicting resources during class loading
> -------------------------------------------------
>
>                 Key: OOZIE-2714
>                 URL: https://issues.apache.org/jira/browse/OOZIE-2714
>             Project: Oozie
>          Issue Type: New Feature
>          Components: core
>            Reporter: Peter Bacsko
>            Assignee: Peter Bacsko
>         Attachments: ClassLoaderTest.java, OOZIE-2714-POC01.patch
>
>
> There are a bunch of issues in Oozie which are related to class loading. 
> The main problem is that the classpath is constructed in a way which is very specific to Oozie:
> - Hadoop lib jars
> - Sharelib jars
> - User-defined jars
> Sometimes there is a conflict between sharelib and hadoop lib version. Also, users can add their own jars which sometimes contain a different version of popular libraries such as Guava, Apache commons, etc.
> We should be able to detect these conflicts and print exact error message so that Oozie users can take appropriate actions to resolve the problem.
> A possible approach is the following:
> * start the execution of an action on a different thread
> * replace the thread's context classloader with a classloader which can detect conflicts
> * when the JVM invokes the {{loadClass()}} method of the classloader, it  scans through the jars (which are available as {{URLClassPath}} objects). If it finds the given resource in at least two jars, it can do different things depending on the setup:
> ** throws an error immediately, mentioning the conflicting jars (this is probably too strict - but still an option)
> ** loads the two resource into a byte array and compares them - it only throws an error if there is difference
> ** compares the jars but only emits an error message if there is a conflict
> ** something else (user defined action?)
> Implementing such a classloader is not difficult and would greatly enhance the supportability of Oozie. It could work in multiple modes depending on the setup - perhaps being able to control it from a workflow config is desirable. If there's any problem, we should be able to turn it off completely, too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)