[jira] Commented: (MAPREDUCE-1938) Ability for having user's classes take precedence over the system classes for tasks' classpath

["Alejandro Abdelnur (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972018#action_12972018 ] 

Alejandro Abdelnur commented on MAPREDUCE-1938:
-----------------------------------------------

IMO this is not the right way of approaching this issue as it will lead to have multiple implementations of a library in the classpath and that sooner or later breaks things in very odd ways. Following an example of an issue I've run into few years ago:

 a-1.jar contains classes: A, B (with B using A)
 a-2.jar contains classes: A, C (with C using A)

A classes from a-1.jar and a-2.jar are different.
Kernel 'sometimes' uses class B from a-1.jar
user-app 'sometimes' uses class C from a-2.jar

All this will compile, but will create very odd errors at runtime.

Because of this, I wouldn't provide a mechanism to allow users to do this.

Instead, I think we should implement a task-classloader that only exposes to the application the MR API and the user is complete control of all the JARs in the app classpath, including input/output formats, etc, etc. 

I understand this can be a bit tricky because the line of API vs implementation classes is not that clear as in the servlet API (to bring an example of APIs with implementations doing this).



> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1938
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: job submission, task, tasktracker
>            Reporter: Devaraj Das
>            Assignee: Krishna Ramachandran
>             Fix For: 0.22.0
>
>         Attachments: mapred-1938-1, mapred-1938-2.patch, mapred-1938-3.patch, mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.