You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Abdul Navaz <na...@gmail.com> on 2014/10/06 02:20:47 UTC
Reduce fails always
Hi All,
I am running sample word count job in a 9 node cluster and I am getting
the below error message.
hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
/user/hduser/getty/out10 -D mapred.reduce.tasks=2
14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
attempt_201410051754_0003_r_000000_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&att
emptid=attempt_201410051754_0003_r_000000_0&filter=stdout
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&att
emptid=attempt_201410051754_0003_r_000000_0&filter=stderr
14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
attempt_201410051754_0003_m_000015_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
I can see it uses all disk space on one of the datanode when shuffling
starts. As soon as disc space on the node becomes nill it throws me this
error and job aborts. The point I didn¹t understand is why it uses only one
datanode disc space. I have change the number of reducer as 4 still it uses
only one datanode disc and throws above error.
How can I fix this issue?
Thanks & Regards,
Navaz
Re: Reduce fails always
Posted by Abdul Navaz <na...@gmail.com>.
It was the space issue. It works perfectly after adding this to mapped.site
xml file.
<property>
<name>mapred.local.dir</name>
<value>/mnt</value>
</property>
Thanks for the help. :)
Thanks & Regards,
Abdul Navaz
Research Assistant
University of Houston Main Campus, Houston TX
Ph: 281-685-0388
From: Ulul <ha...@ulul.org>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, October 6, 2014 at 4:22 PM
To: <us...@hadoop.apache.org>
Subject: Re: Reduce fails always
Hello
Did you check you don't have a job.setNumReduceTasks(1); in your job driver
?
And you should check the number of slots available on the jobtracker web
interface
Ulul
Le 06/10/2014 20:34, Abdul Navaz a écrit :
>
>
> Hello,
>
>
>
>
> I have 8 Datanodes and each having storage capacity of only 3GB. I am running
> word count on 1GB of text file.
>
>
>
>
> Initially df h shows it has 2.8GB after HDFS write. When Shuffling Starts it
> goes on consuming the disc space of only one node. I think it is the reducer.
> Finally df h shows 2MB. Why can¹t it just use all 4 reducer disc space ?
>
>
>
>
>
>
>
>
>
>
>
> Thanks & Regards,
>
>
>
>
> Abdul Navaz
>
> Research Assistant
>
> University of Houston Main Campus, Houston TX
>
> Ph: 281-685-0388
>
>
>
>
>
>
>
>
>
> From: Rohith Sharma K S <ro...@huawei.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Monday, October 6, 2014 at 5:52 AM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: RE: Reduce fails always
>
>
>
>
>
>
>
>
>
> Hi
>
>
>
> How much data does wordcount job is processing?
>
> What is the disk space (³df -h² ) available in the node where it always fail?
>
>
>
> The point I didn¹t understand is why it uses only one datanode disc space?
>
>>> >> For reducers task running, containers can be allocated at any node. I
>>> think, in your cluster one of the machines disk space is very low. So
>>> whichever the task running on that particular node is failing.
>
>
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
>
>
>
>
>
>
>
> From: Abdul Navaz [mailto:navaz.enc@gmail.com]
> Sent: 06 October 2014 08:21
> To: user@hadoop.apache.org
> Subject: Reduce fails always
>
>
>
>
>
>
>
> Hi All,
>
>
>
>
>
>
>
>
>
> I am running sample word count job in a 9 node cluster and I am getting the
> below error message.
>
>
>
>
>
>
>
>
>
> hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
> /user/hduser/getty/out10 -D mapred.reduce.tasks=2
>
>
>
>
>
>
>
>
>
> 14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
>
>
>
>
> 14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
>
>
> 14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
>
>
> 14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
>
>
> 14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_r_000000_0, Status : FAILED
>
>
> FSError: java.io.IOException: No space left on device
>
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attem
> ptid=attempt_201410051754_0003_r_000000_0&filter=stdout
>
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attem
> ptid=attempt_201410051754_0003_r_000000_0&filter=stderr
>
>
> 14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_m_000015_0, Status : FAILED
>
>
> FSError: java.io.IOException: No space left on device
>
>
> 14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
>
>
> 14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> I can see it uses all disk space on one of the datanode when shuffling starts.
> As soon as disc space on the node becomes nill it throws me this error and job
> aborts. The point I didn¹t understand is why it uses only one datanode disc
> space. I have change the number of reducer as 4 still it uses only one
> datanode disc and throws above error.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> How can I fix this issue?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Thanks & Regards,
>
>
>
>
>
>
>
>
>
> Navaz
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Reduce fails always
Posted by Abdul Navaz <na...@gmail.com>.
It was the space issue. It works perfectly after adding this to mapped.site
xml file.
<property>
<name>mapred.local.dir</name>
<value>/mnt</value>
</property>
Thanks for the help. :)
Thanks & Regards,
Abdul Navaz
Research Assistant
University of Houston Main Campus, Houston TX
Ph: 281-685-0388
From: Ulul <ha...@ulul.org>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, October 6, 2014 at 4:22 PM
To: <us...@hadoop.apache.org>
Subject: Re: Reduce fails always
Hello
Did you check you don't have a job.setNumReduceTasks(1); in your job driver
?
And you should check the number of slots available on the jobtracker web
interface
Ulul
Le 06/10/2014 20:34, Abdul Navaz a écrit :
>
>
> Hello,
>
>
>
>
> I have 8 Datanodes and each having storage capacity of only 3GB. I am running
> word count on 1GB of text file.
>
>
>
>
> Initially df h shows it has 2.8GB after HDFS write. When Shuffling Starts it
> goes on consuming the disc space of only one node. I think it is the reducer.
> Finally df h shows 2MB. Why can¹t it just use all 4 reducer disc space ?
>
>
>
>
>
>
>
>
>
>
>
> Thanks & Regards,
>
>
>
>
> Abdul Navaz
>
> Research Assistant
>
> University of Houston Main Campus, Houston TX
>
> Ph: 281-685-0388
>
>
>
>
>
>
>
>
>
> From: Rohith Sharma K S <ro...@huawei.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Monday, October 6, 2014 at 5:52 AM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: RE: Reduce fails always
>
>
>
>
>
>
>
>
>
> Hi
>
>
>
> How much data does wordcount job is processing?
>
> What is the disk space (³df -h² ) available in the node where it always fail?
>
>
>
> The point I didn¹t understand is why it uses only one datanode disc space?
>
>>> >> For reducers task running, containers can be allocated at any node. I
>>> think, in your cluster one of the machines disk space is very low. So
>>> whichever the task running on that particular node is failing.
>
>
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
>
>
>
>
>
>
>
> From: Abdul Navaz [mailto:navaz.enc@gmail.com]
> Sent: 06 October 2014 08:21
> To: user@hadoop.apache.org
> Subject: Reduce fails always
>
>
>
>
>
>
>
> Hi All,
>
>
>
>
>
>
>
>
>
> I am running sample word count job in a 9 node cluster and I am getting the
> below error message.
>
>
>
>
>
>
>
>
>
> hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
> /user/hduser/getty/out10 -D mapred.reduce.tasks=2
>
>
>
>
>
>
>
>
>
> 14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
>
>
>
>
> 14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
>
>
> 14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
>
>
> 14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
>
>
> 14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_r_000000_0, Status : FAILED
>
>
> FSError: java.io.IOException: No space left on device
>
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attem
> ptid=attempt_201410051754_0003_r_000000_0&filter=stdout
>
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attem
> ptid=attempt_201410051754_0003_r_000000_0&filter=stderr
>
>
> 14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_m_000015_0, Status : FAILED
>
>
> FSError: java.io.IOException: No space left on device
>
>
> 14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
>
>
> 14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> I can see it uses all disk space on one of the datanode when shuffling starts.
> As soon as disc space on the node becomes nill it throws me this error and job
> aborts. The point I didn¹t understand is why it uses only one datanode disc
> space. I have change the number of reducer as 4 still it uses only one
> datanode disc and throws above error.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> How can I fix this issue?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Thanks & Regards,
>
>
>
>
>
>
>
>
>
> Navaz
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Reduce fails always
Posted by Abdul Navaz <na...@gmail.com>.
It was the space issue. It works perfectly after adding this to mapped.site
xml file.
<property>
<name>mapred.local.dir</name>
<value>/mnt</value>
</property>
Thanks for the help. :)
Thanks & Regards,
Abdul Navaz
Research Assistant
University of Houston Main Campus, Houston TX
Ph: 281-685-0388
From: Ulul <ha...@ulul.org>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, October 6, 2014 at 4:22 PM
To: <us...@hadoop.apache.org>
Subject: Re: Reduce fails always
Hello
Did you check you don't have a job.setNumReduceTasks(1); in your job driver
?
And you should check the number of slots available on the jobtracker web
interface
Ulul
Le 06/10/2014 20:34, Abdul Navaz a écrit :
>
>
> Hello,
>
>
>
>
> I have 8 Datanodes and each having storage capacity of only 3GB. I am running
> word count on 1GB of text file.
>
>
>
>
> Initially df h shows it has 2.8GB after HDFS write. When Shuffling Starts it
> goes on consuming the disc space of only one node. I think it is the reducer.
> Finally df h shows 2MB. Why can¹t it just use all 4 reducer disc space ?
>
>
>
>
>
>
>
>
>
>
>
> Thanks & Regards,
>
>
>
>
> Abdul Navaz
>
> Research Assistant
>
> University of Houston Main Campus, Houston TX
>
> Ph: 281-685-0388
>
>
>
>
>
>
>
>
>
> From: Rohith Sharma K S <ro...@huawei.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Monday, October 6, 2014 at 5:52 AM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: RE: Reduce fails always
>
>
>
>
>
>
>
>
>
> Hi
>
>
>
> How much data does wordcount job is processing?
>
> What is the disk space (³df -h² ) available in the node where it always fail?
>
>
>
> The point I didn¹t understand is why it uses only one datanode disc space?
>
>>> >> For reducers task running, containers can be allocated at any node. I
>>> think, in your cluster one of the machines disk space is very low. So
>>> whichever the task running on that particular node is failing.
>
>
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
>
>
>
>
>
>
>
> From: Abdul Navaz [mailto:navaz.enc@gmail.com]
> Sent: 06 October 2014 08:21
> To: user@hadoop.apache.org
> Subject: Reduce fails always
>
>
>
>
>
>
>
> Hi All,
>
>
>
>
>
>
>
>
>
> I am running sample word count job in a 9 node cluster and I am getting the
> below error message.
>
>
>
>
>
>
>
>
>
> hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
> /user/hduser/getty/out10 -D mapred.reduce.tasks=2
>
>
>
>
>
>
>
>
>
> 14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
>
>
>
>
> 14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
>
>
> 14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
>
>
> 14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
>
>
> 14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_r_000000_0, Status : FAILED
>
>
> FSError: java.io.IOException: No space left on device
>
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attem
> ptid=attempt_201410051754_0003_r_000000_0&filter=stdout
>
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attem
> ptid=attempt_201410051754_0003_r_000000_0&filter=stderr
>
>
> 14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_m_000015_0, Status : FAILED
>
>
> FSError: java.io.IOException: No space left on device
>
>
> 14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
>
>
> 14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> I can see it uses all disk space on one of the datanode when shuffling starts.
> As soon as disc space on the node becomes nill it throws me this error and job
> aborts. The point I didn¹t understand is why it uses only one datanode disc
> space. I have change the number of reducer as 4 still it uses only one
> datanode disc and throws above error.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> How can I fix this issue?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Thanks & Regards,
>
>
>
>
>
>
>
>
>
> Navaz
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Reduce fails always
Posted by Abdul Navaz <na...@gmail.com>.
It was the space issue. It works perfectly after adding this to mapped.site
xml file.
<property>
<name>mapred.local.dir</name>
<value>/mnt</value>
</property>
Thanks for the help. :)
Thanks & Regards,
Abdul Navaz
Research Assistant
University of Houston Main Campus, Houston TX
Ph: 281-685-0388
From: Ulul <ha...@ulul.org>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, October 6, 2014 at 4:22 PM
To: <us...@hadoop.apache.org>
Subject: Re: Reduce fails always
Hello
Did you check you don't have a job.setNumReduceTasks(1); in your job driver
?
And you should check the number of slots available on the jobtracker web
interface
Ulul
Le 06/10/2014 20:34, Abdul Navaz a écrit :
>
>
> Hello,
>
>
>
>
> I have 8 Datanodes and each having storage capacity of only 3GB. I am running
> word count on 1GB of text file.
>
>
>
>
> Initially df h shows it has 2.8GB after HDFS write. When Shuffling Starts it
> goes on consuming the disc space of only one node. I think it is the reducer.
> Finally df h shows 2MB. Why can¹t it just use all 4 reducer disc space ?
>
>
>
>
>
>
>
>
>
>
>
> Thanks & Regards,
>
>
>
>
> Abdul Navaz
>
> Research Assistant
>
> University of Houston Main Campus, Houston TX
>
> Ph: 281-685-0388
>
>
>
>
>
>
>
>
>
> From: Rohith Sharma K S <ro...@huawei.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Monday, October 6, 2014 at 5:52 AM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: RE: Reduce fails always
>
>
>
>
>
>
>
>
>
> Hi
>
>
>
> How much data does wordcount job is processing?
>
> What is the disk space (³df -h² ) available in the node where it always fail?
>
>
>
> The point I didn¹t understand is why it uses only one datanode disc space?
>
>>> >> For reducers task running, containers can be allocated at any node. I
>>> think, in your cluster one of the machines disk space is very low. So
>>> whichever the task running on that particular node is failing.
>
>
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
>
>
>
>
>
>
>
> From: Abdul Navaz [mailto:navaz.enc@gmail.com]
> Sent: 06 October 2014 08:21
> To: user@hadoop.apache.org
> Subject: Reduce fails always
>
>
>
>
>
>
>
> Hi All,
>
>
>
>
>
>
>
>
>
> I am running sample word count job in a 9 node cluster and I am getting the
> below error message.
>
>
>
>
>
>
>
>
>
> hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
> /user/hduser/getty/out10 -D mapred.reduce.tasks=2
>
>
>
>
>
>
>
>
>
> 14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
>
>
>
>
> 14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
>
>
> 14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
>
>
> 14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
>
>
> 14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_r_000000_0, Status : FAILED
>
>
> FSError: java.io.IOException: No space left on device
>
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attem
> ptid=attempt_201410051754_0003_r_000000_0&filter=stdout
>
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attem
> ptid=attempt_201410051754_0003_r_000000_0&filter=stderr
>
>
> 14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_m_000015_0, Status : FAILED
>
>
> FSError: java.io.IOException: No space left on device
>
>
> 14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
>
>
> 14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> I can see it uses all disk space on one of the datanode when shuffling starts.
> As soon as disc space on the node becomes nill it throws me this error and job
> aborts. The point I didn¹t understand is why it uses only one datanode disc
> space. I have change the number of reducer as 4 still it uses only one
> datanode disc and throws above error.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> How can I fix this issue?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Thanks & Regards,
>
>
>
>
>
>
>
>
>
> Navaz
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Reduce fails always
Posted by Ulul <ha...@ulul.org>.
Hello
Did you check you don't have a job.setNumReduceTasks(1); in your job
driver ?
And you should check the number of slots available on the jobtracker web
interface
Ulul
Le 06/10/2014 20:34, Abdul Navaz a écrit :
> Hello,
>
> I have 8 Datanodes and each having storage capacity of only 3GB. I am
> running word count on 1GB of text file.
>
> Initially df –h shows it has 2.8GB after HDFS write. When Shuffling
> Starts it goes on consuming the disc space of only one node. I think
> it is the reducer. Finally df –h shows 2MB. Why can’t it just use all
> 4 reducer disc space ?
>
>
>
> Thanks & Regards,
>
> Abdul Navaz
> Research Assistant
> University of Houston Main Campus, Houston TX
> Ph: 281-685-0388
>
>
> From: Rohith Sharma K S <rohithsharmaks@huawei.com
> <ma...@huawei.com>>
> Reply-To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Monday, October 6, 2014 at 5:52 AM
> To: "user@hadoop.apache.org <ma...@hadoop.apache.org>"
> <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: RE: Reduce fails always
>
> Hi
>
> How much data does wordcount job is processing?
>
> What is the disk space (“df -h” ) available in the node where it
> always fail?
>
> The point I didn’t understand is why it uses only one datanode disc space?
>
> >> For reducers task running, containers can be allocated at any node. I think, in your cluster one of the
> machines disk space is very low. So whichever the task running on that
> particular node is failing.
>
> Thanks & Regards
>
> Rohith Sharma K S
>
> *From:*Abdul Navaz [mailto:navaz.enc@gmail.com]
> *Sent:* 06 October 2014 08:21
> *To:* user@hadoop.apache.org <ma...@hadoop.apache.org>
> *Subject:* Reduce fails always
>
> Hi All,
>
> I am running sample word count job in a 9 node cluster and I am
> getting the below error message.
>
> hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
> /user/hduser/getty/out10 -D mapred.reduce.tasks=2
>
> 14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
>
> 14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
>
> 14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
>
> 14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
>
> 14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_r_000000_0, Status : FAILED
>
> FSError: java.io.IOException: No space left on device
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stdout
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stderr
>
> 14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_m_000015_0, Status : FAILED
>
> FSError: java.io.IOException: No space left on device
>
> 14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
>
> 14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
>
> I can see it uses all disk space on one of the datanode when shuffling
> starts. As soon as disc space on the node becomes nill it throws me
> this error and job aborts. The point I didn’t understand is why it
> uses only one datanode disc space. I have change the number of
> reducer as 4 still it uses only one datanode disc and throws above error.
>
> How can I fix this issue?
>
> Thanks & Regards,
>
> Navaz
>
Re: Reduce fails always
Posted by Ulul <ha...@ulul.org>.
Hello
Did you check you don't have a job.setNumReduceTasks(1); in your job
driver ?
And you should check the number of slots available on the jobtracker web
interface
Ulul
Le 06/10/2014 20:34, Abdul Navaz a écrit :
> Hello,
>
> I have 8 Datanodes and each having storage capacity of only 3GB. I am
> running word count on 1GB of text file.
>
> Initially df –h shows it has 2.8GB after HDFS write. When Shuffling
> Starts it goes on consuming the disc space of only one node. I think
> it is the reducer. Finally df –h shows 2MB. Why can’t it just use all
> 4 reducer disc space ?
>
>
>
> Thanks & Regards,
>
> Abdul Navaz
> Research Assistant
> University of Houston Main Campus, Houston TX
> Ph: 281-685-0388
>
>
> From: Rohith Sharma K S <rohithsharmaks@huawei.com
> <ma...@huawei.com>>
> Reply-To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Monday, October 6, 2014 at 5:52 AM
> To: "user@hadoop.apache.org <ma...@hadoop.apache.org>"
> <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: RE: Reduce fails always
>
> Hi
>
> How much data does wordcount job is processing?
>
> What is the disk space (“df -h” ) available in the node where it
> always fail?
>
> The point I didn’t understand is why it uses only one datanode disc space?
>
> >> For reducers task running, containers can be allocated at any node. I think, in your cluster one of the
> machines disk space is very low. So whichever the task running on that
> particular node is failing.
>
> Thanks & Regards
>
> Rohith Sharma K S
>
> *From:*Abdul Navaz [mailto:navaz.enc@gmail.com]
> *Sent:* 06 October 2014 08:21
> *To:* user@hadoop.apache.org <ma...@hadoop.apache.org>
> *Subject:* Reduce fails always
>
> Hi All,
>
> I am running sample word count job in a 9 node cluster and I am
> getting the below error message.
>
> hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
> /user/hduser/getty/out10 -D mapred.reduce.tasks=2
>
> 14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
>
> 14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
>
> 14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
>
> 14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
>
> 14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_r_000000_0, Status : FAILED
>
> FSError: java.io.IOException: No space left on device
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stdout
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stderr
>
> 14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_m_000015_0, Status : FAILED
>
> FSError: java.io.IOException: No space left on device
>
> 14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
>
> 14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
>
> I can see it uses all disk space on one of the datanode when shuffling
> starts. As soon as disc space on the node becomes nill it throws me
> this error and job aborts. The point I didn’t understand is why it
> uses only one datanode disc space. I have change the number of
> reducer as 4 still it uses only one datanode disc and throws above error.
>
> How can I fix this issue?
>
> Thanks & Regards,
>
> Navaz
>
Re: Reduce fails always
Posted by Ulul <ha...@ulul.org>.
Hello
Did you check you don't have a job.setNumReduceTasks(1); in your job
driver ?
And you should check the number of slots available on the jobtracker web
interface
Ulul
Le 06/10/2014 20:34, Abdul Navaz a écrit :
> Hello,
>
> I have 8 Datanodes and each having storage capacity of only 3GB. I am
> running word count on 1GB of text file.
>
> Initially df –h shows it has 2.8GB after HDFS write. When Shuffling
> Starts it goes on consuming the disc space of only one node. I think
> it is the reducer. Finally df –h shows 2MB. Why can’t it just use all
> 4 reducer disc space ?
>
>
>
> Thanks & Regards,
>
> Abdul Navaz
> Research Assistant
> University of Houston Main Campus, Houston TX
> Ph: 281-685-0388
>
>
> From: Rohith Sharma K S <rohithsharmaks@huawei.com
> <ma...@huawei.com>>
> Reply-To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Monday, October 6, 2014 at 5:52 AM
> To: "user@hadoop.apache.org <ma...@hadoop.apache.org>"
> <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: RE: Reduce fails always
>
> Hi
>
> How much data does wordcount job is processing?
>
> What is the disk space (“df -h” ) available in the node where it
> always fail?
>
> The point I didn’t understand is why it uses only one datanode disc space?
>
> >> For reducers task running, containers can be allocated at any node. I think, in your cluster one of the
> machines disk space is very low. So whichever the task running on that
> particular node is failing.
>
> Thanks & Regards
>
> Rohith Sharma K S
>
> *From:*Abdul Navaz [mailto:navaz.enc@gmail.com]
> *Sent:* 06 October 2014 08:21
> *To:* user@hadoop.apache.org <ma...@hadoop.apache.org>
> *Subject:* Reduce fails always
>
> Hi All,
>
> I am running sample word count job in a 9 node cluster and I am
> getting the below error message.
>
> hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
> /user/hduser/getty/out10 -D mapred.reduce.tasks=2
>
> 14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
>
> 14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
>
> 14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
>
> 14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
>
> 14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_r_000000_0, Status : FAILED
>
> FSError: java.io.IOException: No space left on device
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stdout
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stderr
>
> 14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_m_000015_0, Status : FAILED
>
> FSError: java.io.IOException: No space left on device
>
> 14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
>
> 14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
>
> I can see it uses all disk space on one of the datanode when shuffling
> starts. As soon as disc space on the node becomes nill it throws me
> this error and job aborts. The point I didn’t understand is why it
> uses only one datanode disc space. I have change the number of
> reducer as 4 still it uses only one datanode disc and throws above error.
>
> How can I fix this issue?
>
> Thanks & Regards,
>
> Navaz
>
Re: Reduce fails always
Posted by Ulul <ha...@ulul.org>.
Hello
Did you check you don't have a job.setNumReduceTasks(1); in your job
driver ?
And you should check the number of slots available on the jobtracker web
interface
Ulul
Le 06/10/2014 20:34, Abdul Navaz a écrit :
> Hello,
>
> I have 8 Datanodes and each having storage capacity of only 3GB. I am
> running word count on 1GB of text file.
>
> Initially df –h shows it has 2.8GB after HDFS write. When Shuffling
> Starts it goes on consuming the disc space of only one node. I think
> it is the reducer. Finally df –h shows 2MB. Why can’t it just use all
> 4 reducer disc space ?
>
>
>
> Thanks & Regards,
>
> Abdul Navaz
> Research Assistant
> University of Houston Main Campus, Houston TX
> Ph: 281-685-0388
>
>
> From: Rohith Sharma K S <rohithsharmaks@huawei.com
> <ma...@huawei.com>>
> Reply-To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Monday, October 6, 2014 at 5:52 AM
> To: "user@hadoop.apache.org <ma...@hadoop.apache.org>"
> <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: RE: Reduce fails always
>
> Hi
>
> How much data does wordcount job is processing?
>
> What is the disk space (“df -h” ) available in the node where it
> always fail?
>
> The point I didn’t understand is why it uses only one datanode disc space?
>
> >> For reducers task running, containers can be allocated at any node. I think, in your cluster one of the
> machines disk space is very low. So whichever the task running on that
> particular node is failing.
>
> Thanks & Regards
>
> Rohith Sharma K S
>
> *From:*Abdul Navaz [mailto:navaz.enc@gmail.com]
> *Sent:* 06 October 2014 08:21
> *To:* user@hadoop.apache.org <ma...@hadoop.apache.org>
> *Subject:* Reduce fails always
>
> Hi All,
>
> I am running sample word count job in a 9 node cluster and I am
> getting the below error message.
>
> hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
> /user/hduser/getty/out10 -D mapred.reduce.tasks=2
>
> 14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
>
> 14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
>
> 14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
>
> 14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
>
> 14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_r_000000_0, Status : FAILED
>
> FSError: java.io.IOException: No space left on device
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stdout
>
> 14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
> outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stderr
>
> 14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
> attempt_201410051754_0003_m_000015_0, Status : FAILED
>
> FSError: java.io.IOException: No space left on device
>
> 14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
>
> 14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
>
> I can see it uses all disk space on one of the datanode when shuffling
> starts. As soon as disc space on the node becomes nill it throws me
> this error and job aborts. The point I didn’t understand is why it
> uses only one datanode disc space. I have change the number of
> reducer as 4 still it uses only one datanode disc and throws above error.
>
> How can I fix this issue?
>
> Thanks & Regards,
>
> Navaz
>
Re: Reduce fails always
Posted by Abdul Navaz <na...@gmail.com>.
Hello,
I have 8 Datanodes and each having storage capacity of only 3GB. I am
running word count on 1GB of text file.
Initially df h shows it has 2.8GB after HDFS write. When Shuffling Starts
it goes on consuming the disc space of only one node. I think it is the
reducer. Finally df h shows 2MB. Why can¹t it just use all 4 reducer disc
space ?
Thanks & Regards,
Abdul Navaz
Research Assistant
University of Houston Main Campus, Houston TX
Ph: 281-685-0388
From: Rohith Sharma K S <ro...@huawei.com>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, October 6, 2014 at 5:52 AM
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: RE: Reduce fails always
Hi
How much data does wordcount job is processing?
What is the disk space (³df -h² ) available in the node where it always
fail?
The point I didn¹t understand is why it uses only one datanode disc space?
>> For reducers task running, containers can be allocated at any node. I think,
in your cluster one of the machines disk space is very low. So whichever the
task running on that particular node is failing.
Thanks & Regards
Rohith Sharma K S
From: Abdul Navaz [mailto:navaz.enc@gmail.com]
Sent: 06 October 2014 08:21
To: user@hadoop.apache.org
Subject: Reduce fails always
Hi All,
I am running sample word count job in a 9 node cluster and I am getting
the below error message.
hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
/user/hduser/getty/out10 -D mapred.reduce.tasks=2
14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
attempt_201410051754_0003_r_000000_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&att
emptid=attempt_201410051754_0003_r_000000_0&filter=stdout
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&att
emptid=attempt_201410051754_0003_r_000000_0&filter=stderr
14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
attempt_201410051754_0003_m_000015_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
I can see it uses all disk space on one of the datanode when shuffling
starts. As soon as disc space on the node becomes nill it throws me this
error and job aborts. The point I didn¹t understand is why it uses only one
datanode disc space. I have change the number of reducer as 4 still it uses
only one datanode disc and throws above error.
How can I fix this issue?
Thanks & Regards,
Navaz
Re: Reduce fails always
Posted by Abdul Navaz <na...@gmail.com>.
Hello,
I have 8 Datanodes and each having storage capacity of only 3GB. I am
running word count on 1GB of text file.
Initially df h shows it has 2.8GB after HDFS write. When Shuffling Starts
it goes on consuming the disc space of only one node. I think it is the
reducer. Finally df h shows 2MB. Why can¹t it just use all 4 reducer disc
space ?
Thanks & Regards,
Abdul Navaz
Research Assistant
University of Houston Main Campus, Houston TX
Ph: 281-685-0388
From: Rohith Sharma K S <ro...@huawei.com>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, October 6, 2014 at 5:52 AM
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: RE: Reduce fails always
Hi
How much data does wordcount job is processing?
What is the disk space (³df -h² ) available in the node where it always
fail?
The point I didn¹t understand is why it uses only one datanode disc space?
>> For reducers task running, containers can be allocated at any node. I think,
in your cluster one of the machines disk space is very low. So whichever the
task running on that particular node is failing.
Thanks & Regards
Rohith Sharma K S
From: Abdul Navaz [mailto:navaz.enc@gmail.com]
Sent: 06 October 2014 08:21
To: user@hadoop.apache.org
Subject: Reduce fails always
Hi All,
I am running sample word count job in a 9 node cluster and I am getting
the below error message.
hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
/user/hduser/getty/out10 -D mapred.reduce.tasks=2
14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
attempt_201410051754_0003_r_000000_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&att
emptid=attempt_201410051754_0003_r_000000_0&filter=stdout
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&att
emptid=attempt_201410051754_0003_r_000000_0&filter=stderr
14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
attempt_201410051754_0003_m_000015_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
I can see it uses all disk space on one of the datanode when shuffling
starts. As soon as disc space on the node becomes nill it throws me this
error and job aborts. The point I didn¹t understand is why it uses only one
datanode disc space. I have change the number of reducer as 4 still it uses
only one datanode disc and throws above error.
How can I fix this issue?
Thanks & Regards,
Navaz
Re: Reduce fails always
Posted by Abdul Navaz <na...@gmail.com>.
Hello,
I have 8 Datanodes and each having storage capacity of only 3GB. I am
running word count on 1GB of text file.
Initially df h shows it has 2.8GB after HDFS write. When Shuffling Starts
it goes on consuming the disc space of only one node. I think it is the
reducer. Finally df h shows 2MB. Why can¹t it just use all 4 reducer disc
space ?
Thanks & Regards,
Abdul Navaz
Research Assistant
University of Houston Main Campus, Houston TX
Ph: 281-685-0388
From: Rohith Sharma K S <ro...@huawei.com>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, October 6, 2014 at 5:52 AM
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: RE: Reduce fails always
Hi
How much data does wordcount job is processing?
What is the disk space (³df -h² ) available in the node where it always
fail?
The point I didn¹t understand is why it uses only one datanode disc space?
>> For reducers task running, containers can be allocated at any node. I think,
in your cluster one of the machines disk space is very low. So whichever the
task running on that particular node is failing.
Thanks & Regards
Rohith Sharma K S
From: Abdul Navaz [mailto:navaz.enc@gmail.com]
Sent: 06 October 2014 08:21
To: user@hadoop.apache.org
Subject: Reduce fails always
Hi All,
I am running sample word count job in a 9 node cluster and I am getting
the below error message.
hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
/user/hduser/getty/out10 -D mapred.reduce.tasks=2
14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
attempt_201410051754_0003_r_000000_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&att
emptid=attempt_201410051754_0003_r_000000_0&filter=stdout
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&att
emptid=attempt_201410051754_0003_r_000000_0&filter=stderr
14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
attempt_201410051754_0003_m_000015_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
I can see it uses all disk space on one of the datanode when shuffling
starts. As soon as disc space on the node becomes nill it throws me this
error and job aborts. The point I didn¹t understand is why it uses only one
datanode disc space. I have change the number of reducer as 4 still it uses
only one datanode disc and throws above error.
How can I fix this issue?
Thanks & Regards,
Navaz
Re: Reduce fails always
Posted by Abdul Navaz <na...@gmail.com>.
Hello,
I have 8 Datanodes and each having storage capacity of only 3GB. I am
running word count on 1GB of text file.
Initially df h shows it has 2.8GB after HDFS write. When Shuffling Starts
it goes on consuming the disc space of only one node. I think it is the
reducer. Finally df h shows 2MB. Why can¹t it just use all 4 reducer disc
space ?
Thanks & Regards,
Abdul Navaz
Research Assistant
University of Houston Main Campus, Houston TX
Ph: 281-685-0388
From: Rohith Sharma K S <ro...@huawei.com>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, October 6, 2014 at 5:52 AM
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: RE: Reduce fails always
Hi
How much data does wordcount job is processing?
What is the disk space (³df -h² ) available in the node where it always
fail?
The point I didn¹t understand is why it uses only one datanode disc space?
>> For reducers task running, containers can be allocated at any node. I think,
in your cluster one of the machines disk space is very low. So whichever the
task running on that particular node is failing.
Thanks & Regards
Rohith Sharma K S
From: Abdul Navaz [mailto:navaz.enc@gmail.com]
Sent: 06 October 2014 08:21
To: user@hadoop.apache.org
Subject: Reduce fails always
Hi All,
I am running sample word count job in a 9 node cluster and I am getting
the below error message.
hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt
/user/hduser/getty/out10 -D mapred.reduce.tasks=2
14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
14/10/05 18:08:58 INFO mapred.JobClient: Task Id :
attempt_201410051754_0003_r_000000_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&att
emptid=attempt_201410051754_0003_r_000000_0&filter=stdout
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task
outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&att
emptid=attempt_201410051754_0003_r_000000_0&filter=stderr
14/10/05 18:08:59 INFO mapred.JobClient: Task Id :
attempt_201410051754_0003_m_000015_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
I can see it uses all disk space on one of the datanode when shuffling
starts. As soon as disc space on the node becomes nill it throws me this
error and job aborts. The point I didn¹t understand is why it uses only one
datanode disc space. I have change the number of reducer as 4 still it uses
only one datanode disc and throws above error.
How can I fix this issue?
Thanks & Regards,
Navaz
RE: Reduce fails always
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi
How much data does wordcount job is processing?
What is the disk space ("df -h" ) available in the node where it always fail?
The point I didn't understand is why it uses only one datanode disc space?
>> For reducers task running, containers can be allocated at any node. I think, in your cluster one of the machines disk space is very low. So whichever the task running on that particular node is failing.
Thanks & Regards
Rohith Sharma K S
From: Abdul Navaz [mailto:navaz.enc@gmail.com]
Sent: 06 October 2014 08:21
To: user@hadoop.apache.org
Subject: Reduce fails always
Hi All,
I am running sample word count job in a 9 node cluster and I am getting the below error message.
hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt /user/hduser/getty/out10 -D mapred.reduce.tasks=2
14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
14/10/05 18:08:58 INFO mapred.JobClient: Task Id : attempt_201410051754_0003_r_000000_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stdout
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stderr
14/10/05 18:08:59 INFO mapred.JobClient: Task Id : attempt_201410051754_0003_m_000015_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
I can see it uses all disk space on one of the datanode when shuffling starts. As soon as disc space on the node becomes nill it throws me this error and job aborts. The point I didn't understand is why it uses only one datanode disc space. I have change the number of reducer as 4 still it uses only one datanode disc and throws above error.
How can I fix this issue?
Thanks & Regards,
Navaz
RE: Reduce fails always
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi
How much data does wordcount job is processing?
What is the disk space ("df -h" ) available in the node where it always fail?
The point I didn't understand is why it uses only one datanode disc space?
>> For reducers task running, containers can be allocated at any node. I think, in your cluster one of the machines disk space is very low. So whichever the task running on that particular node is failing.
Thanks & Regards
Rohith Sharma K S
From: Abdul Navaz [mailto:navaz.enc@gmail.com]
Sent: 06 October 2014 08:21
To: user@hadoop.apache.org
Subject: Reduce fails always
Hi All,
I am running sample word count job in a 9 node cluster and I am getting the below error message.
hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt /user/hduser/getty/out10 -D mapred.reduce.tasks=2
14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
14/10/05 18:08:58 INFO mapred.JobClient: Task Id : attempt_201410051754_0003_r_000000_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stdout
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stderr
14/10/05 18:08:59 INFO mapred.JobClient: Task Id : attempt_201410051754_0003_m_000015_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
I can see it uses all disk space on one of the datanode when shuffling starts. As soon as disc space on the node becomes nill it throws me this error and job aborts. The point I didn't understand is why it uses only one datanode disc space. I have change the number of reducer as 4 still it uses only one datanode disc and throws above error.
How can I fix this issue?
Thanks & Regards,
Navaz
RE: Reduce fails always
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi
How much data does wordcount job is processing?
What is the disk space ("df -h" ) available in the node where it always fail?
The point I didn't understand is why it uses only one datanode disc space?
>> For reducers task running, containers can be allocated at any node. I think, in your cluster one of the machines disk space is very low. So whichever the task running on that particular node is failing.
Thanks & Regards
Rohith Sharma K S
From: Abdul Navaz [mailto:navaz.enc@gmail.com]
Sent: 06 October 2014 08:21
To: user@hadoop.apache.org
Subject: Reduce fails always
Hi All,
I am running sample word count job in a 9 node cluster and I am getting the below error message.
hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt /user/hduser/getty/out10 -D mapred.reduce.tasks=2
14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
14/10/05 18:08:58 INFO mapred.JobClient: Task Id : attempt_201410051754_0003_r_000000_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stdout
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stderr
14/10/05 18:08:59 INFO mapred.JobClient: Task Id : attempt_201410051754_0003_m_000015_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
I can see it uses all disk space on one of the datanode when shuffling starts. As soon as disc space on the node becomes nill it throws me this error and job aborts. The point I didn't understand is why it uses only one datanode disc space. I have change the number of reducer as 4 still it uses only one datanode disc and throws above error.
How can I fix this issue?
Thanks & Regards,
Navaz
RE: Reduce fails always
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi
How much data does wordcount job is processing?
What is the disk space ("df -h" ) available in the node where it always fail?
The point I didn't understand is why it uses only one datanode disc space?
>> For reducers task running, containers can be allocated at any node. I think, in your cluster one of the machines disk space is very low. So whichever the task running on that particular node is failing.
Thanks & Regards
Rohith Sharma K S
From: Abdul Navaz [mailto:navaz.enc@gmail.com]
Sent: 06 October 2014 08:21
To: user@hadoop.apache.org
Subject: Reduce fails always
Hi All,
I am running sample word count job in a 9 node cluster and I am getting the below error message.
hadoop jar chiu-wordcount2.jar WordCount /user/hduser/getty/file1.txt /user/hduser/getty/out10 -D mapred.reduce.tasks=2
14/10/05 18:08:45 INFO mapred.JobClient: map 99% reduce 26%
14/10/05 18:08:48 INFO mapred.JobClient: map 99% reduce 28%
14/10/05 18:08:51 INFO mapred.JobClient: map 100% reduce 28%
14/10/05 18:08:57 INFO mapred.JobClient: map 98% reduce 0%
14/10/05 18:08:58 INFO mapred.JobClient: Task Id : attempt_201410051754_0003_r_000000_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stdout
14/10/05 18:08:59 WARN mapred.JobClient: Error reading task outputhttp://pcvm1-10.utahddc.geniracks.net:50060/tasklog?plaintext=true&attemptid=attempt_201410051754_0003_r_000000_0&filter=stderr
14/10/05 18:08:59 INFO mapred.JobClient: Task Id : attempt_201410051754_0003_m_000015_0, Status : FAILED
FSError: java.io.IOException: No space left on device
14/10/05 18:09:02 INFO mapred.JobClient: map 99% reduce 0%
14/10/05 18:09:07 INFO mapred.JobClient: map 99% reduce 1%
I can see it uses all disk space on one of the datanode when shuffling starts. As soon as disc space on the node becomes nill it throws me this error and job aborts. The point I didn't understand is why it uses only one datanode disc space. I have change the number of reducer as 4 still it uses only one datanode disc and throws above error.
How can I fix this issue?
Thanks & Regards,
Navaz