You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jim Colestock <jc...@ramblingredneck.com> on 2013/08/30 15:47:48 UTC
reduce job hung in pending state: "No room for reduce task"
Hello All,
We're running into the following 2 bugs again:
https://issues.apache.org/jira/browse/HADOOP-5241
https://issues.apache.org/jira/browse/MAPREDUCE-2324
Both of them a listed as closed fixed. (I was actually the one that got cloudera to submit MAPREDUCE-2324) Does anyone know is anyone else seeing these in later releases? We're running the following on various version of Cent OS with Java 1.6:
hadoop-2.0.0+1357-1.cdh4.3.0.p0.21.el5
hadoop-0.20-mapreduce-jobtracker-2.0.0+1357-1.cdh4.3.0.p0.21.el5
hadoop-0.20-mapreduce-2.0.0+1357-1.cdh4.3.0.p0.21.el5
hadoop-0.20-mapreduce-tasktracker-2.0.0+1357-1.cdh4.3.0.p0.21.el5
hadoop-hdfs-namenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
hadoop-hdfs-secondarynamenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
hadoop-hdfs-2.0.0+1357-1.cdh4.3.0.p0.21.el5
hadoop-hdfs-datanode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
Just for a quick summary, basically a reduce job get hung in pending while trying to find room on a task tracker, it keeps trying over and over and never fails. So you end up with a whole bunch of these in the logs:
2013-08-27 00:48:01,412 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_104.sm.tld:127.0.0.1/127.0.0.1:43723 has 250176954368 bytes free; but we expect reduce input to take 283580756533
Thanks in advance for any help on the issue..
JC
Re: reduce job hung in pending state: "No room for reduce task"
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
Did you checked the free disk space on server where your reducer task
was running?
because it need approx. 264gb free disk space to run(as per logs).
Thanks
Jitendra
On 8/30/13, Jim Colestock <jc...@ramblingredneck.com> wrote:
> Hello All,
>
> We're running into the following 2 bugs again:
> https://issues.apache.org/jira/browse/HADOOP-5241
> https://issues.apache.org/jira/browse/MAPREDUCE-2324
>
> Both of them a listed as closed fixed. (I was actually the one that got
> cloudera to submit MAPREDUCE-2324) Does anyone know is anyone else seeing
> these in later releases? We're running the following on various version of
> Cent OS with Java 1.6:
>
> hadoop-2.0.0+1357-1.cdh4.3.0.p0.21.el5
>
> hadoop-0.20-mapreduce-jobtracker-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-0.20-mapreduce-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-0.20-mapreduce-tasktracker-2.0.0+1357-1.cdh4.3.0.p0.21.el5
>
> hadoop-hdfs-namenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-hdfs-secondarynamenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-hdfs-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-hdfs-datanode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
>
> Just for a quick summary, basically a reduce job get hung in pending while
> trying to find room on a task tracker, it keeps trying over and over and
> never fails. So you end up with a whole bunch of these in the logs:
>
> 2013-08-27 00:48:01,412 WARN org.apache.hadoop.mapred.JobInProgress: No room
> for reduce task. Node tracker_104.sm.tld:127.0.0.1/127.0.0.1:43723 has
> 250176954368 bytes free; but we expect reduce input to take 283580756533
>
> Thanks in advance for any help on the issue..
>
> JC
>
>
>
>
>
>
Re: reduce job hung in pending state: "No room for reduce task"
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
Did you checked the free disk space on server where your reducer task
was running?
because it need approx. 264gb free disk space to run(as per logs).
Thanks
Jitendra
On 8/30/13, Jim Colestock <jc...@ramblingredneck.com> wrote:
> Hello All,
>
> We're running into the following 2 bugs again:
> https://issues.apache.org/jira/browse/HADOOP-5241
> https://issues.apache.org/jira/browse/MAPREDUCE-2324
>
> Both of them a listed as closed fixed. (I was actually the one that got
> cloudera to submit MAPREDUCE-2324) Does anyone know is anyone else seeing
> these in later releases? We're running the following on various version of
> Cent OS with Java 1.6:
>
> hadoop-2.0.0+1357-1.cdh4.3.0.p0.21.el5
>
> hadoop-0.20-mapreduce-jobtracker-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-0.20-mapreduce-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-0.20-mapreduce-tasktracker-2.0.0+1357-1.cdh4.3.0.p0.21.el5
>
> hadoop-hdfs-namenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-hdfs-secondarynamenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-hdfs-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-hdfs-datanode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
>
> Just for a quick summary, basically a reduce job get hung in pending while
> trying to find room on a task tracker, it keeps trying over and over and
> never fails. So you end up with a whole bunch of these in the logs:
>
> 2013-08-27 00:48:01,412 WARN org.apache.hadoop.mapred.JobInProgress: No room
> for reduce task. Node tracker_104.sm.tld:127.0.0.1/127.0.0.1:43723 has
> 250176954368 bytes free; but we expect reduce input to take 283580756533
>
> Thanks in advance for any help on the issue..
>
> JC
>
>
>
>
>
>
Re: reduce job hung in pending state: "No room for reduce task"
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
Did you checked the free disk space on server where your reducer task
was running?
because it need approx. 264gb free disk space to run(as per logs).
Thanks
Jitendra
On 8/30/13, Jim Colestock <jc...@ramblingredneck.com> wrote:
> Hello All,
>
> We're running into the following 2 bugs again:
> https://issues.apache.org/jira/browse/HADOOP-5241
> https://issues.apache.org/jira/browse/MAPREDUCE-2324
>
> Both of them a listed as closed fixed. (I was actually the one that got
> cloudera to submit MAPREDUCE-2324) Does anyone know is anyone else seeing
> these in later releases? We're running the following on various version of
> Cent OS with Java 1.6:
>
> hadoop-2.0.0+1357-1.cdh4.3.0.p0.21.el5
>
> hadoop-0.20-mapreduce-jobtracker-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-0.20-mapreduce-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-0.20-mapreduce-tasktracker-2.0.0+1357-1.cdh4.3.0.p0.21.el5
>
> hadoop-hdfs-namenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-hdfs-secondarynamenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-hdfs-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-hdfs-datanode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
>
> Just for a quick summary, basically a reduce job get hung in pending while
> trying to find room on a task tracker, it keeps trying over and over and
> never fails. So you end up with a whole bunch of these in the logs:
>
> 2013-08-27 00:48:01,412 WARN org.apache.hadoop.mapred.JobInProgress: No room
> for reduce task. Node tracker_104.sm.tld:127.0.0.1/127.0.0.1:43723 has
> 250176954368 bytes free; but we expect reduce input to take 283580756533
>
> Thanks in advance for any help on the issue..
>
> JC
>
>
>
>
>
>
Re: reduce job hung in pending state: "No room for reduce task"
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
Did you checked the free disk space on server where your reducer task
was running?
because it need approx. 264gb free disk space to run(as per logs).
Thanks
Jitendra
On 8/30/13, Jim Colestock <jc...@ramblingredneck.com> wrote:
> Hello All,
>
> We're running into the following 2 bugs again:
> https://issues.apache.org/jira/browse/HADOOP-5241
> https://issues.apache.org/jira/browse/MAPREDUCE-2324
>
> Both of them a listed as closed fixed. (I was actually the one that got
> cloudera to submit MAPREDUCE-2324) Does anyone know is anyone else seeing
> these in later releases? We're running the following on various version of
> Cent OS with Java 1.6:
>
> hadoop-2.0.0+1357-1.cdh4.3.0.p0.21.el5
>
> hadoop-0.20-mapreduce-jobtracker-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-0.20-mapreduce-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-0.20-mapreduce-tasktracker-2.0.0+1357-1.cdh4.3.0.p0.21.el5
>
> hadoop-hdfs-namenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-hdfs-secondarynamenode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-hdfs-2.0.0+1357-1.cdh4.3.0.p0.21.el5
> hadoop-hdfs-datanode-2.0.0+1357-1.cdh4.3.0.p0.21.el5
>
> Just for a quick summary, basically a reduce job get hung in pending while
> trying to find room on a task tracker, it keeps trying over and over and
> never fails. So you end up with a whole bunch of these in the logs:
>
> 2013-08-27 00:48:01,412 WARN org.apache.hadoop.mapred.JobInProgress: No room
> for reduce task. Node tracker_104.sm.tld:127.0.0.1/127.0.0.1:43723 has
> 250176954368 bytes free; but we expect reduce input to take 283580756533
>
> Thanks in advance for any help on the issue..
>
> JC
>
>
>
>
>
>