You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by George Pang <p0...@gmail.com> on 2009/07/21 19:03:07 UTC
Spill failed error
Hi users,
Please help with this one - I got an error at running a two - node cluster
on big files, the error is :
2365222 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher -
Error message from task (map)
task_200907201602_0002_m_000010*java.io.IOException:
Spill failed*
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:425)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:96)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:81)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2198)
*Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not
find any valid local directory for
taskTracker/jobcache/job_200907201602_0002/attempt_200907201602_0002_m_000010_0/output/spill0.out
*
*Cluster specification: *
Hadoop -version 0.18.3
Ubuntu -version 8.04
*etc/hosts*
127.0.0.1 localhost
127.0.1.1 wpang-laptop
192.168.0.1 master
192.168.0.2 slave
*hadoop-site.xml*
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-datastore/hadoop-${user.name}</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>master:9001</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
*Hadoop web interface ( http://master:50070)
* Cluster Summary * * * 54 files and directories, 196 blocks = 250 total.
Heap Size is 20.5 MB / 963 MB (2%)
* Capacity : 54.49 GB DFS Remaining : 18.21 GB DFS Used : 22.59 GB DFS
Used% : 41.45 % Live Nodes <http://localhost:50070/dfshealth.jsp#LiveNodes>
: 2 Dead Nodes <http://localhost:50070/dfshealth.jsp#DeadNodes> : 0
George
Re: Spill failed error
Posted by Vibhooti Verma <ve...@gmail.com>.
This mainly happens when you do not have enough space. Please clean up and
run again.
On Tue, Jul 21, 2009 at 10:33 PM, George Pang <p0...@gmail.com> wrote:
> Hi users,
>
> Please help with this one - I got an error at running a two - node cluster
> on big files, the error is :
>
> 2365222 [main] ERROR
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher -
> Error message from task (map)
> task_200907201602_0002_m_000010*java.io.IOException:
> Spill failed*
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:425)
> at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:96)
> at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172)
> at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158)
> at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:81)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2198)
> *Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
> not
> find any valid local directory for
>
> taskTracker/jobcache/job_200907201602_0002/attempt_200907201602_0002_m_000010_0/output/spill0.out
> *
>
> *Cluster specification: *
>
> Hadoop -version 0.18.3
> Ubuntu -version 8.04
>
> *etc/hosts*
>
> 127.0.0.1 localhost
> 127.0.1.1 wpang-laptop
> 192.168.0.1 master
> 192.168.0.2 slave
>
> *hadoop-site.xml*
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/usr/local/hadoop-datastore/hadoop-${user.name}</value>
> </property>
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://master:9000</value>
> </property>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>master:9001</value>
> </property>
>
> <property>
> <name>dfs.replication</name>
> <value>2</value>
> </property>
>
>
> *Hadoop web interface ( http://master:50070)
> * Cluster Summary * * * 54 files and directories, 196 blocks = 250 total.
> Heap Size is 20.5 MB / 963 MB (2%)
> * Capacity : 54.49 GB DFS Remaining : 18.21 GB DFS Used : 22.59 GB DFS
> Used% : 41.45 % Live Nodes <http://localhost:50070/dfshealth.jsp#LiveNodes
> >
> : 2 Dead Nodes <http://localhost:50070/dfshealth.jsp#DeadNodes> : 0
> George
>
--
cheers,
Vibhooti