[jira] [Commented] (MAPREDUCE-6108) ShuffleError OOM while reserving memory by MergeManagerImpl

["Xuan Gong (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14328415#comment-14328415 ] 

Xuan Gong commented on MAPREDUCE-6108:
--------------------------------------

[~kasha] Thanks for the investigation. It makes sense for me. I have meet the same issue here.

But, we got the  "java.lang.OutOfMemoryError: Java heap space" here. Does it mean we are running out of jvm heap memory instead of "memoryLimit" ?
So, even we still use "if (usedMemory > memoryLimit)" condition, and use all the default configuration (memoryLimit = 0.7 * maxMemory). In the worse case, usedMemory = memoryLimit = 0.7 * maxMemory, requestedSize = 0.25*0.7*maxMemory. So, total will be 0.875 maxMemory. I guess that is why we use " (usedMemory > memoryLimit)" instead of "(usedMemory + requestedSize > memoryLimit)" here. 

I guess that we might not have a big issue on logic itself. In the same JVM, not only the shuffle need memory, but also other staff occupy some part of the memory (such as fetch thread ?). I guess this OOM happens when we are trying to use "too-many" memory for the shuffle(such as the worst case happens).

I think for now, this is more related to the configuration expertise.
*mapreduce.reduce.shuffle.input.buffer.percent
*mapreduce.reduce.shuffle.memory.limit.percent
*mapreduce.reduce.shuffle.parallelcopies

Thoughts ?

> ShuffleError OOM while reserving memory by MergeManagerImpl
> -----------------------------------------------------------
>
>                 Key: MAPREDUCE-6108
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6108
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.5.1
>            Reporter: Dongwook Kwon
>            Priority: Critical
>
> Shuffle has OOM issue from time to time.  
> Such as this email reported.
> http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-dev/201408.mbox/%3CCABWXXjNK-on0XTrMuriJD8SDGJjTAMSvQW2CZpm3oEkJ3YM8YQ@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)