[jira] [Updated] (MAPREDUCE-5785) Derive task attempt JVM max heap size and io.sort.mb automatically from mapreduce.*.memory.mb

["Karthik Kambatla (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Karthik Kambatla updated MAPREDUCE-5785:
----------------------------------------
    Attachment: mr-5785-4.patch

Uploading a patch that does the following:
# If -Xmx is not set in the conf, it calculates this from the memory.mb value.
# If memory.mb is not set in the conf, it calculates this from the -Xmx value.
# If neither are set, it uses the default memory.mb value and calculates -Xmx from this.
# Updates a bunch of descriptions to mapred-default.xml to capture this clearly. Let me know if they are not clear enough.
# A little bit of test cleanup. 

The patch reverts changes to io.sort.mb from earlier versions to be addressed in a follow-up JIRA.

To parse the -Xmx value from java-opts, the patch "borrows" [~jeagles] patch from TEZ-1508. [~jeagles] - hope you are fine with that. 
 

> Derive task attempt JVM max heap size and io.sort.mb automatically from mapreduce.*.memory.mb
> ---------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5785
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mr-am, task
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5785.v01.patch, MAPREDUCE-5785.v02.patch, MAPREDUCE-5785.v03.patch, mr-5785-4.patch
>
>
> Currently users have to set 2 memory-related configs per Job / per task type.  One first chooses some container size map reduce.\*.memory.mb and then a corresponding maximum Java heap size Xmx < map reduce.\*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be 
> - allocating big containers whereas the JVM will only use the default -Xmx200m.
> - allocating small containers that will OOM because Xmx is too high.
> With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)