You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by wangxiaowei <wa...@software.ict.ac.cn> on 2008/05/13 17:12:36 UTC
why it stopped at Reduce phase?
hi all:
I uses two computers A and B as a hadoop cluster,A is JobTracker and NameNode,both A and B are slaves.
The input data size is about 80MB,including 100,000records. The job is to read one record a time and find some useful content in it,and transmit it to reduce.
But when I submit it ,it just run map tasks,and the reduce task did not run at all!This is the JobTacker`s log file:
2008-05-13 21:02:00,007 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805132055_0001_m_000000_0: FSError: java.io.IOException: No space left on device
2008-05-13 21:02:11,952 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805132055_0001_m_000001_0: FSError: java.io.IOException: No space left on device
2008-05-13 21:02:11,953 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805132055_0001_m_000006_0: FSError: java.io.IOException: No space left on device
I think it is something wrong with the configuration file,Can you give me some suggestions?
Do you meet the same questions?
Re: why it stopped at Reduce phase?
Posted by James Moore <ja...@gmail.com>.
Also, hadoop.tmp.dir defaults to storing files in
/tmp/hadoop-{username} - it's very common for machines to not have a
lot of space available in /tmp. You'll probably want to change that
location.
--
James Moore | james@restphone.com
blog.restphone.com
Re: why it stopped at Reduce phase?
Posted by Arun C Murthy <ar...@yahoo-inc.com>.
Wang,
On May 13, 2008, at 8:12 AM, wangxiaowei wrote:
> hi all:
> I uses two computers A and B as a hadoop cluster,A is JobTracker
> and NameNode,both A and B are slaves.
> The input data size is about 80MB,including 100,000records. The job
> is to read one record a time and find some useful content in it,and
> transmit it to reduce.
> But when I submit it ,it just run map tasks,and the reduce task
> did not run at all!This is the JobTacker`s log file:
> 2008-05-13 21:02:00,007 INFO
> org.apache.hadoop.mapred.TaskInProgress: Error from
> task_200805132055_0001_m_000000_0: FSError: java.io.IOException: No
> space left on device
> 2008-05-13 21:02:11,952 INFO
> org.apache.hadoop.mapred.TaskInProgress: Error from
> task_200805132055_0001_m_000001_0: FSError: java.io.IOException: No
> space left on device
> 2008-05-13 21:02:11,953 INFO
> org.apache.hadoop.mapred.TaskInProgress: Error from
> task_200805132055_0001_m_000006_0: FSError: java.io.IOException: No
> space left on device
> I think it is something wrong with the configuration file,Can
> you give me some suggestions?
> Do you meet the same questions?
It looks like your maps are failing since there isn't enough
temporary disk space on your nodes.
Usually the outputs of maps are stored on the local-disk of the nodes
on which they were executed, and it looks like your nodes don't have
enough.
You can use multiple disks for your intermediate map-outputs by using
the configuration knob: *mapred.local.dir*
(http://hadoop.apache.org/core/docs/current/hadoop-
default.html#mapred.local.dir).
Arun