[jira] [Commented] (MAPREDUCE-4301) Dedupe some strings in MRAM for memory savings

["Robert Joseph Evans (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287610#comment-13287610 ] 

Robert Joseph Evans commented on MAPREDUCE-4301:
------------------------------------------------

This patch is a small safe one that only covers the JobSplit.

I looked at several other strings that also had a lot of duplicates.  

TaskAttemptStatusUpdateEvent$TaskAttemptState.stateString
MapTaskAttemptImpl.diagnostics
The keys to Counters.groups
GenericGroup.displayName
The keys to GenericGroup.counters
and GenericCounter.displayName

None of the rest of these were as simple to solve because they all seem to ultimate result from either incoming RPC messages or from a parsed file.  The only way to fix these is to try and dedupe them on the fly.  This can be done with String.intern, but at the risk of filling up the permgen space.  Or with playing games with our own internal cache. 

Either of these solutions is much more risky then just a two line change so I will file a separate JIRA to explore those options.
                
> Dedupe some strings in MRAM for memory savings
> ----------------------------------------------
>
>                 Key: MAPREDUCE-4301
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4301
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster
>    Affects Versions: 0.23.3, 2.0.0-alpha
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>         Attachments: MR-4301.txt
>
>
> Recently an OutOfMemoryError caused one of our jobs to become a zombie (MAPREDUCE-4300).  It was a rather large job with 78000+ map tasks and only 750MB of heap configured.  I took a heap dump to see if there were any obvious memory leaks, and I could not find any, but yourkit and some digging found some potential memory optimizations that we could do.
> In this particular case we could save about 20MB if SplitMetaInfoReader.readSplitMetaInfo only computed the JobSplitFile once instead of for each split. (a 2 line change)
> I will look into some others and see if there are more savings I can come up with.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira