[jira] Commented: (MAPREDUCE-967) TaskTracker does not need to fully unjar job jars

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753455#action_12753455 ] 

Todd Lipcon commented on MAPREDUCE-967:
---------------------------------------

Currently, TaskTracker.localizeJob completely unjars job.jar in jobCacheDir. TaskRunner then appends <jobCacheDir>/classes:<jobCacheDir>/lib/*:<jobCacheDir>/ to the task classpath. Instead, I propose that we only unpack the classes/ and lib/ portions of job.jar, and add <jobCacheDir>/job.jar to the task classpath in lieu of <jobCacheDir>/

While we're at it, I'm not sure I see the purpose of the "classes/" directory - this is not standard Jar layout by any means, and seems unnecessary. But that issue is orthogonal to this ticket.

Attaching a preliminary patch against branch-20, though this should go into trunk and probably not the branch. I just want to test this on a real workload first.

> TaskTracker does not need to fully unjar job jars
> -------------------------------------------------
>
>                 Key: MAPREDUCE-967
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-967
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>
> In practice we have seen some users submitting job jars that consist of 10,000+ classes. Unpacking these jars into mapred.local.dir and then cleaning up after them has a significant cost (both in wall clock and in unnecessary heavy disk utilization). This cost can be easily avoided

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.