You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Henning Blohm (JIRA)" <ji...@apache.org> on 2010/11/04 14:35:48 UTC

[jira] Commented: (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928208#action_12928208 ] 

Henning Blohm commented on MAPREDUCE-1700:
------------------------------------------

The patch in MAPREDUCE-1938 does unfortunately not solve the issue when the job implementation uses custom class loaders to load dependency classes. The proposed patch only addresses the issue when no custom class loaders are in the picture.

As a first step, it would really help (us anyway), if jobs would not be started with stuff on the classpath that is not at all required for job execution per se (e.g. jetty libs, the eclipse java compiler, jasper...).

Secondly, hadoop could actually start with only the hadoop api types on the classpath plus a small launcher that would load hadoops implementation in an isolated child loader, so that implementation dependencies do not leak through to the job's implementation. I am not sure if the hadoop implementation is ready for implementation/api separation via class loaders though. 

I patched hadoop 0.20.2 to exclude all libs in lib/server from the jobs classpath and I move all non-job related jars into that server folder in my hadoop installation. That helped somewhat. 

> User supplied dependencies may conflict with MapReduce system JARs
> ------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1700
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>            Reporter: Tom White
>
> If user code has a dependency on a version of a JAR that is different to the one that happens to be used by Hadoop, then it may not work correctly. This happened with user code using a different version of Avro, as reported [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852081#action_12852081].
> The problem is analogous to the one that application servers have with WAR loading. Using a specialized classloader in the Child JVM is probably the way to solve this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.