Hive - MR Join TaskAttemptListenerImpl +Spill failed

[fasi meg <fa...@yahoo.com>]

I have multiple tables, and was trying to recursively join them. The first join finishes with no error but not the second 

i am not sure if it have something to do using hive tables as input to the next join... because every time I use external tables that i created there is no error at all, it finishes the job.

in hive.log


2013-08-30 17:06:32,145 WARN  shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(48)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode.


I set this property to fix this problem but still no change...
 <property>

     <name>hive.exec.show.job.failure.debug.info</name>
     <value>false</value>
 </property>

task tracker log
2013-08-30 18:01:58,200 FATAL [IPC Server handler 25 on 47684] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1377878075746_0001_m_000025_0 - exited : java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {............}
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {.............}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:565)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Spill failed
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:304)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
... 9 more
Caused by: java.io.IOException: Spill failed
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTask.java:1489)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1011)
at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:550)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:289)
... 15 more
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1377878075746_0001_m_000025_0_spill_1.out
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:385)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFiles.java:159)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1522)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$600(MapTask.java:813)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1459)

the syntax i used is


CREATE TABLE Join_Two ROW FORMAT DELIMITED FIELDS TERMINATED BY '9' LINES TERMINATED BY '\n' STORED AS TEXTFILE as select /*+MAPJOIN(JOIN_TWO)*/ * From Join_one fact LEFT OUTER JOIN  Dim2 dim WHERE fact.id = dim.id;


I am using
Hadoop 0.23.1
Hive 0.10.0
MR2/Yarn

I have 9 data nodes and 1 name node. The tables are not big tables, I am sure it is not a size problem.