You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Adam Sotona (JIRA)" <ji...@apache.org> on 2017/06/07 06:50:18 UTC
[jira] [Commented] (MAPREDUCE-6447) reduce shuffle throws
"java.lang.OutOfMemoryError: Java heap space"
[ https://issues.apache.org/jira/browse/MAPREDUCE-6447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040315#comment-16040315 ]
Adam Sotona commented on MAPREDUCE-6447:
----------------------------------------
Hi guys,
I'm experiencing exactly the same issue, even I decreased the values following way:
mapreduce.reduce.shuffle.input.buffer.percent=0.70
mapreduce.reduce.shuffle.memory.limit.percent=0.2
mapreduce.reduce.shuffle.parallelcopies=4
According to the above calculations 0.7 * 0.2 * 4 = 0.56 < 1
Could someone explain it?
I'm running very long MapReduce process, where some of the reducers almost deterministically fail on OOME, repeated attempts still cause OOME and the whole restarted application with reduced shuffle parameters still fails (after several days of computation).
Is there any reason why InMemoryMapOutput blindly allocates memory despite the actual situation?
Could Fetcher.copyMapOutput recover from that situation by falling into an alternative MapOutput?
Thanks
> reduce shuffle throws "java.lang.OutOfMemoryError: Java heap space"
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-6447
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6447
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.5.0, 2.6.0, 2.5.1, 2.7.1
> Reporter: shuzhangyao
> Assignee: shuzhangyao
> Priority: Minor
>
> 2015-08-11 14:03:54,550 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#10
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
> at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
> at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)
> at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:303)
> at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:293)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:511)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:329)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org