[jira] Commented: (MAPREDUCE-1478) Separate the mapred.lib and mapreduce.lib classes to a different jar and include the user jar ahead of the lib jar.

["Scott Carey (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983334#action_12983334 ] 

Scott Carey commented on MAPREDUCE-1478:
----------------------------------------

This approach will work.

I see another possibility that is simpler, but has some serious drawbacks.

Instead of separating the source directories and classpaths at build time, file set includes/excludes could be used at jar creation time to create two jar files.
Benefit:  simpler, requires no code changes:
Drawback:  does not actually disentangle the code, which is more brittle and could lead to confusion depending on what changes are in the user lib jars.

That is a big drawback.  

The above approach is more complicated but does enforce true separation.
In the long run, tackling the disentangling of the lib from the system is a requirement but the extra complexity requires more careful review.  It also is the bulk of the work required to make the lib a truly separate library -- for maven or OSGi.

I'm not an expert on the details of the API and code changes needed for clean separation as a consequence of this.  The callback changes seem like an improvement.  Duplication of wrappers and reflective access are usually signs that there is a better way, but that better way almost always requires an API break.

Overall, I like this approach because it is moving in the right direction for separating the core execution from more volatile or user-extensible library code. 

> Separate the mapred.lib and mapreduce.lib classes to a different jar and include the user jar ahead of the lib jar.
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1478
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1478
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: task
>            Reporter: Owen O'Malley
>         Attachments: MAPREDUCE-1478.patch, move.sh
>
>
> Currently the user can't include updated library jars as part of their job. By pulling out the lib classes we can include the classes (eg. TextInputFormat) in the user's jar and get their version and not the system installed one.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.