[jira] [Resolved] (MAPREDUCE-2872) Optimize TaskTracker memory usage

["Allen Wittenauer (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer resolved MAPREDUCE-2872.
-----------------------------------------
    Resolution: Won't Fix

Closing this as won't fix, given development on hadoop-1 has effectively stopped.

> Optimize TaskTracker memory usage
> ---------------------------------
>
>                 Key: MAPREDUCE-2872
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2872
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: tasktracker
>    Affects Versions: 0.20.203.0
>            Reporter: Binglin Chang
>            Assignee: Binglin Chang
>              Labels: memory, optimization
>
> We observe high memory usage of framework level components on slave node, mainly TaskTracker & Child, especially for large clusters. To be clear at first, large jobs with 10000-100000 map and >10000 reduce tasks are very common in our offline cluster, and will very likely continue to grow. This is reasonable because the number of map & reduce slots are in the same range, and it's impractical for users to reduce their job's task number without execution time penalty. 
> High memory consumption will:
> * Limit the memory used by up level application; 
> * Reduce page cache space, which plays a  important role in spill, merge, shuffle and even HDFS performance; 
> * Increase the probability of slave node OOM, which may affect storage layer(HDFS) too. 
> A stable TT with predictable memory behavior is desired, this also applies to Child JVM.
> This issue focuses on TaskTracker memory optimization, on our cluster, TaskTracker use 600M+ memory & 300%+(3core+) CPU at peak, and 300M+ memory & much less CPU in average, so we need to set -Xmx to 1000M for TT to prevent OOM, then the TT memory is in 200M-1200M range, and 800M in average. 
> Here are some ideas:  
> Jetty http connection use a lot memory when these are many requests in queue, we need to limit the length of the queue, combine multiple requests into one request, or use netty just like MR2
> TaskCompletionEvents use a lot memory too if a job have large number of map task, this won't be a problem in MR2, but can be optimized, A typical TaskCompletionEvent object use 296 bytes memory, a job with 100000 map will use about 30M memory, problem will appear if there are some big RunningJob in a TaskTracker. There are more memory efficient implementations for TaskCompletionEvent.
> IndexCache: memory of indexcache varies directly as reduce number, on large cluster 10MB of indexcache is not enough, 
> we set it to 100MB, again use primitive long[] instead of IndexRecord[] can save 50% of memory.
> Although some of the above won't be a problem in MR-v2, since MR-v1 is still widely used, I think optimizations are needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)