You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hadoop.apache.org by - <co...@ymail.com> on 2013/10/25 22:20:30 UTC

Reduce Merge Memtomem Parameter

Hi All,

What do "mapreduce.reduce.merge.memtomem.threshold" and "mapreduce.reduce.merge.memtomem.enabled" do and by default is it enabled? There's no documentation for them but I noticed that they exist in hadoop 2.2.0.

Thanks!

Re: Reduce Merge Memtomem Parameter

Posted by Ravi Prakash <ra...@ymail.com>.

Hi!

This parameter triggers a sort of fetched map outputs on the reducer node when the number of in memory map outputs > memToMemMergeOutputsThreshold . It is disabled by default. I am guessing this was put in on the premise that it might be faster to 
sort fewer number of streams even in memory. And also we can sort 
perhaps while waiting to fetch.

HTH
Ravi.




On Friday, October 25, 2013 3:22 PM, - <co...@ymail.com> wrote:
 
Hi All,

What do "mapreduce.reduce.merge.memtomem.threshold" and "mapreduce.reduce.merge.memtomem.enabled" do and by default is it enabled? There's no documentation for them but I noticed that they exist in hadoop 2.2.0.

Thanks!

Re: Reduce Merge Memtomem Parameter

Posted by Ravi Prakash <ra...@ymail.com>.

Hi!

This parameter triggers a sort of fetched map outputs on the reducer node when the number of in memory map outputs > memToMemMergeOutputsThreshold . It is disabled by default. I am guessing this was put in on the premise that it might be faster to 
sort fewer number of streams even in memory. And also we can sort 
perhaps while waiting to fetch.

HTH
Ravi.




On Friday, October 25, 2013 3:22 PM, - <co...@ymail.com> wrote:
 
Hi All,

What do "mapreduce.reduce.merge.memtomem.threshold" and "mapreduce.reduce.merge.memtomem.enabled" do and by default is it enabled? There's no documentation for them but I noticed that they exist in hadoop 2.2.0.

Thanks!

Re: Reduce Merge Memtomem Parameter

Posted by Ravi Prakash <ra...@ymail.com>.

Hi!

This parameter triggers a sort of fetched map outputs on the reducer node when the number of in memory map outputs > memToMemMergeOutputsThreshold . It is disabled by default. I am guessing this was put in on the premise that it might be faster to 
sort fewer number of streams even in memory. And also we can sort 
perhaps while waiting to fetch.

HTH
Ravi.




On Friday, October 25, 2013 3:22 PM, - <co...@ymail.com> wrote:
 
Hi All,

What do "mapreduce.reduce.merge.memtomem.threshold" and "mapreduce.reduce.merge.memtomem.enabled" do and by default is it enabled? There's no documentation for them but I noticed that they exist in hadoop 2.2.0.

Thanks!

Re: Reduce Merge Memtomem Parameter

Posted by Ravi Prakash <ra...@ymail.com>.

Hi!

This parameter triggers a sort of fetched map outputs on the reducer node when the number of in memory map outputs > memToMemMergeOutputsThreshold . It is disabled by default. I am guessing this was put in on the premise that it might be faster to 
sort fewer number of streams even in memory. And also we can sort 
perhaps while waiting to fetch.

HTH
Ravi.




On Friday, October 25, 2013 3:22 PM, - <co...@ymail.com> wrote:
 
Hi All,

What do "mapreduce.reduce.merge.memtomem.threshold" and "mapreduce.reduce.merge.memtomem.enabled" do and by default is it enabled? There's no documentation for them but I noticed that they exist in hadoop 2.2.0.

Thanks!