You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Wilfred Spiegelenburg (JIRA)" <ji...@apache.org> on 2017/08/31 03:57:00 UTC

[jira] [Commented] (MAPREDUCE-6447) reduce shuffle throws "java.lang.OutOfMemoryError: Java heap space"

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148409#comment-16148409 ] 

Wilfred Spiegelenburg commented on MAPREDUCE-6447:
--------------------------------------------------

[~prateekrungta] and [~shuzhangyao]: I have seen this same issue a number of times and people keep referring to this open MR issue.
I have dug into this and have found that there is nothing wrong with the calculation and there is no need for a change in the way we handle this in the code. There is no guarantee that any method devised for the internal calculation will guarantee that you do not get an OOM error. In all cases I have run into I have been able to fix it with a configuration change in MR and the JVM.

Let me explain what I have found and why the issue will not be solved by changing the internal calculations.

When the JVM throws an OOM for a reducer I collected a heap dumps and looked at what was allocated at the point in time that the OOM was thrown. It most cases the OOM was not thrown due to the total heap being used. As an example: the JVM heap for the reducer was set to 9216MB or 9GB the heap dump showed only a 5896M heap usage. Looking at the usage of the heap it showed that the shuffle input buffer usage was well within its limits.
We then tried to lower the {{mapreduce.reduce.shuffle.input.buffer.percent}} from the default 0.9 to 0.6 and found that it did not solve the issue. There was still an OOM around the same point with approximately the same usage of the heap. Lowering it further to 0.4 allowed the job to finish but we saw that the JVM never peaked above about 60% of the assigned heap. This causes a lot of waste on the cluster and is thus not a solution we could accept.
Further checks of the GC logging showed that all heap usage was in the old generation for each of the OOM cases. That explains why the reducer was throwing an OOM and the heap dump: it had run out of space in the old generation, not the total heap. Within the heap the old generation can take about 2/3 of the total heap. This is based on the default settings for the generation sizing in the heap [1].

The question then became what caused the JVM to run out of old generation and not using its young generation? This could be explained by the logging from the reducer we had. The reducer logs showed that it was trying to allocate a large shuffle response. In my case about 1.9GB. Eventhough this is a large shuffle response it was within all the limit. The JVM will allocate large objects often directly in the old generation instead of allocating it in the young generation. This behaviour could cause an OOM error in the reducer while not using the full heap and just running out of old generation.

Back to the calculations. In the buffer we load all the shuffle data but we set a maximum of 25% of the total buffer in one shuffle response. This is the in memory merge limit. If the shuffle response is larger than 25% of the buffer size we do not store it in the buffer but directly merge to disk. A shuffle response is only accepted and downloaded if we can fit it in memory or if it goes straight to disk. The check and increase of the buffer usage happens before we start the download. Locking makes sure only one thread does this at a time, the number of paralel copies is thus not important. Limiting could lead to a deadlock as explained in the comments in the code. Since we need to prevent deadlocks we allow one shuffle (one thread) to go over that limit. If we would not allow this we could deadlock the reducer. The reducer would be in a state that it can not download new data to reduce. There would also never be a trigger that would cause the data in memory to be merged/spilled and thus the buffer stays as full as it is.

Based on all that the maximum size of all the data we could store in the shuffle buffer would be:
{code}
mapreduce.reduce.shuffle.input.buffer.percent = buffer% = 70%
mapreduce.reduce.shuffle.memory.limit.percent = limit% = 25%
heap size = 9GB
maximum used memory = ((buffer% * (1 + limit%)) * heap size) - 1 byte
{code}
If that buffer does not fit in the old generation we could throw an OOM error without really running out of memory. This is especially true when the individual shuffle sizes are large but not hit the in memory limit. Everything is still properly calculated and limited. We also do not unknowningly use more than the configured buffer size. If we go over we know exactly how much.

The way we worked around the problem without increasing the size of the heap. We did this by changing the generations. The old generation inside the heap was changed by increasing the "NewRatio" setting from 2 (default) to 4. We also changed the "input.buffer.percent" setting to 65%. That worked in our case with the 9GB as the maximum heap for the reducer. Different heap sizes combined with a different "memory.limit.percent" might require a different "Newratio" setting.

[1] https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/sizing.html

> reduce shuffle throws "java.lang.OutOfMemoryError: Java heap space"
> -------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6447
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6447
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.5.0, 2.6.0, 2.5.1, 2.7.1
>            Reporter: shuzhangyao
>            Assignee: shuzhangyao
>            Priority: Minor
>
> 2015-08-11 14:03:54,550 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#10
> 	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 	at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
> 	at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
> 	at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)
> 	at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:303)
> 	at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:293)
> 	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:511)
> 	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:329)
> 	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org