You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Kang Xiao (JIRA)" <ji...@apache.org> on 2011/03/01 02:02:38 UTC
[jira] Commented: (MAPREDUCE-2345) Optimize jobtracker's memory
usage
[ https://issues.apache.org/jira/browse/MAPREDUCE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000637#comment-13000637 ]
Kang Xiao commented on MAPREDUCE-2345:
--------------------------------------
Thanks Arun, it's really a good solution to retire completed jobs from memory and the function is in trunk. But how about a running job with tens of thousands of tasks? We see that big running jobs use much memory in the cluster.
> Optimize jobtracker's memory usage
> -------------------------------------
>
> Key: MAPREDUCE-2345
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2345
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: MengWang
> Labels: hadoop
> Fix For: 0.23.0
>
> Attachments: jt-memory-useage.bmp
>
>
> To many tasks will eat up a considerable amount of JobTracker's heap space. According to our observation, 50GB heap size can support to 5,000,000 tasks, so we should optimize jobtracker's memory usage for more jobs and tasks. Yourkit java profile show that counters, duplicate strings, Task waste too much memory. Our optimization around these three points reduced jobtracker's memory to 1/3.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira