You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-dev@hadoop.apache.org by Radim Kolar <hs...@filez.com> on 2013/05/06 04:35:58 UTC

oom during shuffle

i retested it and oom error is also in 2.0.3, but not in 1.x.

how much memory is needed for reducer during shuffle and sort? Peak 
usage is memoryLimit or there have to be additional memory for 
mapred.reduce.parallel.copies fetch buffers?

MergerManager: memoryLimit=326264416, maxSingleShuffleLimit=81566104, 
mergeThreshold=215334528, ioSortFactor=10, memToMemMergeOutputsThreshold=10