[jira] Commented: (MAPREDUCE-1374) Reduce memory footprint of FileSplit

["Tom White (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848519#action_12848519 ] 

Tom White commented on MAPREDUCE-1374:
--------------------------------------

bq. Since String.intern() takes up space in the PermGen area, JobClient should not get OOM because of the low PermGen heap space of JobClient. What should we do about it? A current client with low PermGen space and trying to submit a job with large input splits will fail with this patch.

Good point. One approach would be to use a "weak object pool", which uses the heap rather than PermGen, as described here: http://www.codeinstructions.com/2008/09/instance-pools-with-weakhashmap.html

> Reduce memory footprint of FileSplit
> ------------------------------------
>
>                 Key: MAPREDUCE-1374
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1374
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: MAPREDUCE-1374.1.patch, MAPREDUCE-1374.2.patch, MAPREDUCE-1374.3.patch
>
>
> We can have many FileInput objects in the memory, depending on the number of mappers.
> It will save tons of memory on JobTracker and JobClient if we intern those Strings for host names.
> {code}
> FileInputFormat.java:
>       for (NodeInfo host: hostList) {
>         // Strip out the port number from the host name
> -        retVal[index++] = host.node.getName().split(":")[0];
> +        retVal[index++] = host.node.getName().split(":")[0].intern();
>         if (index == replicationFactor) {
>           done = true;
>           break;
>         }
>       }
> {code}
> More on String.intern(): http://www.javaworld.com/javaworld/javaqa/2003-12/01-qa-1212-intern.html
> It will also save a lot of memory by changing the class of {{file}} from {{Path}} to {{String}}. {{Path}} contains a {{java.net.URI}} which internally contains ~10 String fields. This will also be a huge saving.
> {code}
>   private Path file;
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.