You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Stephen Boesch <ja...@gmail.com> on 2012/07/04 17:46:58 UTC
Job exceeded Reduce Input limit
I am running a (terasort) job on a small cluster but with powerful nodes.
The number of reducer slots was 12. I am seeing the following message:
Job JOBID="job_201207031814_0011" FINISH_TIME="1341389866650"
JOB_STATUS="FAILED" FINISHED_MAPS="42" FINISHED_REDUCES="0"
FAIL_REASON="Job exceeded Reduce Input limit Limit: 10737418240
Estimated: 102000004905" .
Now this apparently was added recently:
http://mail-archives.apache.org/mod_mbox/hadoop-common-commits/201103.mbox/%3C20110304042718.5854E23888CD@eris.apache.org%3E
It looks that the solution would be to set mapreduce.reduce.input.limit to
-1:
<property>
+ <name>mapreduce.reduce.input.limit</name>
+ <value>-1</value>
+ <description>The limit on the input size of the reduce. If the estimated
+ input size of the reduce is greater than this value, job is failed. A
+ value of -1 means that there is no limit set. </description>
+</property>
I did that (in mapred-site.xml). But it did not affect the behavior
i.e. the problem continues.
Any hints appreciated.
thx!
Re: Job exceeded Reduce Input limit
Posted by Marcos Ortiz <ml...@uci.cu>.
Can you share with us your completed stack?
- Java and Hadoop version?
- Config files?
- Cluster distribution? (Numbers of NN and DN)
- O.S
If it is possible, your hardware too.
Best wishes
On 07/06/2012 10:57 AM, Robert Evans wrote:
> What version of hadoop are you using?
>
> From: Stephen Boesch <javadba@gmail.com <ma...@gmail.com>>
> Reply-To: "mapreduce-user@hadoop.apache.org
> <ma...@hadoop.apache.org>"
> <mapreduce-user@hadoop.apache.org
> <ma...@hadoop.apache.org>>
> To: "mapreduce-user@hadoop.apache.org
> <ma...@hadoop.apache.org>"
> <mapreduce-user@hadoop.apache.org
> <ma...@hadoop.apache.org>>
> Subject: Job exceeded Reduce Input limit
>
>
> I am running a (terasort) job on a small cluster but with powerful
> nodes. The number of reducer slots was 12. I am seeing the following
> message:
> Job JOBID="job_201207031814_0011" FINISH_TIME="1341389866650"
> JOB_STATUS="FAILED" FINISHED_MAPS="42" FINISHED_REDUCES="0"
> FAIL_REASON="Job exceeded Reduce Input limit Limit: 10737418240
> Estimated: 102000004905" .
>
>
> Now this apparently was added recently:
>
> http://mail-archives.apache.org/mod_mbox/hadoop-common-commits/201103.mbox/%3C20110304042718.5854E23888CD@eris.apache.org%3E
>
>
> It looks that the solution would be to set
> mapreduce.reduce.input.limit to -1:
>
>
> <property>
> + <name>mapreduce.reduce.input.limit</name>
> + <value>-1</value>
> + <description>The limit on the input size of the reduce. If the estimated
> + input size of the reduce is greater than this value, job is failed. A
> + value of -1 means that there is no limit set. </description>
> +</property>
> I did that (in mapred-site.xml). But it did not affect the behavior i.e. the problem continues.
> Any hints appreciated.
> thx!
>
>
> <http://www.uci.cu/>
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Re: Job exceeded Reduce Input limit
Posted by Robert Evans <ev...@yahoo-inc.com>.
What version of hadoop are you using?
From: Stephen Boesch <ja...@gmail.com>>
Reply-To: "mapreduce-user@hadoop.apache.org<ma...@hadoop.apache.org>" <ma...@hadoop.apache.org>>
To: "mapreduce-user@hadoop.apache.org<ma...@hadoop.apache.org>" <ma...@hadoop.apache.org>>
Subject: Job exceeded Reduce Input limit
I am running a (terasort) job on a small cluster but with powerful nodes. The number of reducer slots was 12. I am seeing the following message:
Job JOBID="job_201207031814_0011" FINISH_TIME="1341389866650" JOB_STATUS="FAILED" FINISHED_MAPS="42" FINISHED_REDUCES="0" FAIL_REASON="Job exceeded Reduce Input limit Limit: 10737418240 Estimated: 102000004905" .
Now this apparently was added recently:
http://mail-archives.apache.org/mod_mbox/hadoop-common-commits/201103.mbox/%3C20110304042718.5854E23888CD@eris.apache.org%3E
It looks that the solution would be to set mapreduce.reduce.input.limit to -1:
<property>
+ <name>mapreduce.reduce.input.limit</name>
+ <value>-1</value>
+ <description>The limit on the input size of the reduce. If the estimated
+ input size of the reduce is greater than this value, job is failed. A
+ value of -1 means that there is no limit set. </description>
+</property>
I did that (in mapred-site.xml). But it did not affect the behavior i.e. the problem continues.
Any hints appreciated.
thx!