You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Elia Mazzawi <el...@casalemedia.com> on 2008/09/29 21:45:20 UTC

Could not find any valid local directory for task_200809041356_0042_r_000000_2/intermediate.9

what does this exception mean?

08/09/27 07:56:11 INFO mapred.JobClient: Task Id : 
task_200809041356_0042_r_000000_2, Status : FAILED
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find 
any valid local directory for 
task_200809041356_0042_r_000000_2/intermediate.9
        at 
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
        at 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
        at 
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2851)
        at 
org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2586)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:352)
        at 
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)

Re: Could not find any valid local directory for task_200809041356_0042_r_000000_2/intermediate.9

Posted by Miles Osborne <mi...@inf.ed.ac.uk>.
check that you are not getting disk full errors

Miles

2008/9/29 Elia Mazzawi <el...@casalemedia.com>:
> in more detail, my program is happily chugging along until the reducer fails
> with that exception, then it looks like it retries and fails by itself.
> the same hadoop program works fine on a subset of the data.
> I'm rerunning on all the subsets to see if there is anything in the data
> that is causing this,
>
> but can someone explain what this error means,
> I'm runnig hadoop 17.0 maybe its time to update
>
>
> 08/09/27 05:59:01 INFO mapred.JobClient:  map 96% reduce 31%
> 08/09/27 06:02:17 INFO mapred.JobClient:  map 96% reduce 32%
> 08/09/27 06:24:46 INFO mapred.JobClient:  map 97% reduce 32%
> 08/09/27 06:49:38 INFO mapred.JobClient:  map 98% reduce 32%
> 08/09/27 07:14:12 INFO mapred.JobClient:  map 99% reduce 32%
> 08/09/27 07:17:09 INFO mapred.JobClient:  map 99% reduce 33%
> 08/09/27 07:37:50 INFO mapred.JobClient:  map 100% reduce 33%
> 08/09/27 07:56:11 INFO mapred.JobClient:  map 100% reduce 0%
> 08/09/27 07:56:11 INFO mapred.JobClient: Task Id :
> task_200809041356_0042_r_000000_2, Status : FAILED
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for task_200809041356_0042_r_000000_2/intermediate.9
>       at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
>       at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
>       at
> org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2851)
>       at
> org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2586)
>       at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:352)
>       at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
>
> 08/09/27 07:57:10 INFO mapred.JobClient:  map 100% reduce 1%
> 08/09/27 07:57:55 INFO mapred.JobClient:  map 100% reduce 2%
> 08/09/27 07:58:46 INFO mapred.JobClient:  map 100% reduce 3%
> 08/09/27 07:59:36 INFO mapred.JobClient:  map 100% reduce 4%
> 08/09/27 08:00:26 INFO mapred.JobClient:  map 100% reduce 5%
> 08/09/27 08:01:16 INFO mapred.JobClient:  map 100% reduce 6%
> 08/09/27 08:02:06 INFO mapred.JobClient:  map 100% reduce 7%
> 08/09/27 08:02:55 INFO mapred.JobClient:  map 100% reduce 8%
> 08/09/27 08:03:45 INFO mapred.JobClient:  map 100% reduce 9%
> 08/09/27 08:04:36 INFO mapred.JobClient:  map 100% reduce 10%
> 08/09/27 08:05:26 INFO mapred.JobClient:  map 100% reduce 11%
> 08/09/27 08:06:18 INFO mapred.JobClient:  map 100% reduce 12%
> 08/09/27 08:07:09 INFO mapred.JobClient:  map 100% reduce 13%
> 08/09/27 08:08:00 INFO mapred.JobClient:  map 100% reduce 14%
> 08/09/27 08:08:50 INFO mapred.JobClient:  map 100% reduce 15%
> 08/09/27 08:09:45 INFO mapred.JobClient:  map 100% reduce 16%
> 08/09/27 08:10:31 INFO mapred.JobClient:  map 100% reduce 17%
> 08/09/27 08:11:26 INFO mapred.JobClient:  map 100% reduce 18%
> 08/09/27 08:12:16 INFO mapred.JobClient:  map 100% reduce 19%
> 08/09/27 08:13:08 INFO mapred.JobClient:  map 100% reduce 20%
> 08/09/27 08:14:02 INFO mapred.JobClient:  map 100% reduce 21%
> 08/09/27 08:14:50 INFO mapred.JobClient:  map 100% reduce 22%
> 08/09/27 08:15:41 INFO mapred.JobClient:  map 100% reduce 23%
> 08/09/27 08:16:36 INFO mapred.JobClient:  map 100% reduce 24%
> 08/09/27 08:17:26 INFO mapred.JobClient:  map 100% reduce 25%
> 08/09/27 08:18:14 INFO mapred.JobClient:  map 100% reduce 26%
> 08/09/27 08:19:02 INFO mapred.JobClient:  map 100% reduce 27%
> 08/09/27 08:19:55 INFO mapred.JobClient:  map 100% reduce 28%
> 08/09/27 08:21:12 INFO mapred.JobClient:  map 100% reduce 71%
> java.io.IOException: Job failed!
>       at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1062)
>       at org.myorg.binAnalysis.main(binAnalysis.java:99)
>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>       at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>       at java.lang.reflect.Method.invoke(Method.java:585)
>       at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
>       at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
>       at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>       at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>       at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
>
> You have new mail in /var/spool/mail/root
>
>
> Elia Mazzawi wrote:
>>
>> what does this exception mean?
>>
>> 08/09/27 07:56:11 INFO mapred.JobClient: Task Id :
>> task_200809041356_0042_r_000000_2, Status : FAILED
>> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
>> valid local directory for task_200809041356_0042_r_000000_2/intermediate.9
>>       at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
>>       at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
>>       at
>> org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2851)
>>       at
>> org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2586)
>>       at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:352)
>>       at
>> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
>>
>
>



-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

Re: Could not find any valid local directory for task_200809041356_0042_r_000000_2/intermediate.9

Posted by Elia Mazzawi <el...@casalemedia.com>.
in more detail, my program is happily chugging along until the reducer 
fails with that exception, then it looks like it retries and fails by 
itself.
the same hadoop program works fine on a subset of the data.
I'm rerunning on all the subsets to see if there is anything in the data 
that is causing this,

but can someone explain what this error means,
I'm runnig hadoop 17.0 maybe its time to update


08/09/27 05:59:01 INFO mapred.JobClient:  map 96% reduce 31%
08/09/27 06:02:17 INFO mapred.JobClient:  map 96% reduce 32%
08/09/27 06:24:46 INFO mapred.JobClient:  map 97% reduce 32%
08/09/27 06:49:38 INFO mapred.JobClient:  map 98% reduce 32%
08/09/27 07:14:12 INFO mapred.JobClient:  map 99% reduce 32%
08/09/27 07:17:09 INFO mapred.JobClient:  map 99% reduce 33%
08/09/27 07:37:50 INFO mapred.JobClient:  map 100% reduce 33%
08/09/27 07:56:11 INFO mapred.JobClient:  map 100% reduce 0%
08/09/27 07:56:11 INFO mapred.JobClient: Task Id : 
task_200809041356_0042_r_000000_2, Status : FAILED
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find 
any valid local directory for 
task_200809041356_0042_r_000000_2/intermediate.9
        at 
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
        at 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
        at 
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2851)
        at 
org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2586)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:352)
        at 
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)

08/09/27 07:57:10 INFO mapred.JobClient:  map 100% reduce 1%
08/09/27 07:57:55 INFO mapred.JobClient:  map 100% reduce 2%
08/09/27 07:58:46 INFO mapred.JobClient:  map 100% reduce 3%
08/09/27 07:59:36 INFO mapred.JobClient:  map 100% reduce 4%
08/09/27 08:00:26 INFO mapred.JobClient:  map 100% reduce 5%
08/09/27 08:01:16 INFO mapred.JobClient:  map 100% reduce 6%
08/09/27 08:02:06 INFO mapred.JobClient:  map 100% reduce 7%
08/09/27 08:02:55 INFO mapred.JobClient:  map 100% reduce 8%
08/09/27 08:03:45 INFO mapred.JobClient:  map 100% reduce 9%
08/09/27 08:04:36 INFO mapred.JobClient:  map 100% reduce 10%
08/09/27 08:05:26 INFO mapred.JobClient:  map 100% reduce 11%
08/09/27 08:06:18 INFO mapred.JobClient:  map 100% reduce 12%
08/09/27 08:07:09 INFO mapred.JobClient:  map 100% reduce 13%
08/09/27 08:08:00 INFO mapred.JobClient:  map 100% reduce 14%
08/09/27 08:08:50 INFO mapred.JobClient:  map 100% reduce 15%
08/09/27 08:09:45 INFO mapred.JobClient:  map 100% reduce 16%
08/09/27 08:10:31 INFO mapred.JobClient:  map 100% reduce 17%
08/09/27 08:11:26 INFO mapred.JobClient:  map 100% reduce 18%
08/09/27 08:12:16 INFO mapred.JobClient:  map 100% reduce 19%
08/09/27 08:13:08 INFO mapred.JobClient:  map 100% reduce 20%
08/09/27 08:14:02 INFO mapred.JobClient:  map 100% reduce 21%
08/09/27 08:14:50 INFO mapred.JobClient:  map 100% reduce 22%
08/09/27 08:15:41 INFO mapred.JobClient:  map 100% reduce 23%
08/09/27 08:16:36 INFO mapred.JobClient:  map 100% reduce 24%
08/09/27 08:17:26 INFO mapred.JobClient:  map 100% reduce 25%
08/09/27 08:18:14 INFO mapred.JobClient:  map 100% reduce 26%
08/09/27 08:19:02 INFO mapred.JobClient:  map 100% reduce 27%
08/09/27 08:19:55 INFO mapred.JobClient:  map 100% reduce 28%
08/09/27 08:21:12 INFO mapred.JobClient:  map 100% reduce 71%
java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1062)
        at org.myorg.binAnalysis.main(binAnalysis.java:99)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)

You have new mail in /var/spool/mail/root


Elia Mazzawi wrote:
> what does this exception mean?
>
> 08/09/27 07:56:11 INFO mapred.JobClient: Task Id : 
> task_200809041356_0042_r_000000_2, Status : FAILED
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find 
> any valid local directory for 
> task_200809041356_0042_r_000000_2/intermediate.9
>        at 
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313) 
>
>        at 
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) 
>
>        at 
> org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2851) 
>
>        at 
> org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2586)
>        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:352)
>        at 
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
>