You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Oren Bumgarner <or...@gmail.com> on 2013/04/26 20:50:30 UTC
Exception during reduce phase when running jobs remotely
I have a small hadoop cluster running 1.0.4 and I'm trying to have it setup
so that I can run jobs remotely from a computer on the same network that is
not a part of the cluster. I've got a main java class that
implements org.apache.hadoop.util.Tool and I'm able to run this job from
the NameNode using ToolRunner.run(), setting up the JobConf, and submitting
with JobClient.submitJob().
When I try to run the same class remotely from any machine that is not the
NameNode the job is submitted and it appears that the Map tasks
successfully complete, but I get the following exception for all of the
reduce tasks:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
output/map_0.out in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.MapOutputFile.getInputFile(MapOutputFile.java:161)
at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:220)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:398)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
I'm not sure how to interpret that error message. I think I'm missing
some config files that are not present on the remote machine but I
can't pin down exactly what I need. Does anyone have any guidance on
what the above error means or how to submit jobs remotely?
Thanks,
--Oren
Re: Exception during reduce phase when running jobs remotely
Posted by sudhakara st <su...@gmail.com>.
Hello,
Check you hadoop.tmp.dir and mapred.local.dir configuration and permissions.
On Sat, Apr 27, 2013 at 12:20 AM, Oren Bumgarner <or...@gmail.com> wrote:
> I have a small hadoop cluster running 1.0.4 and I'm trying to have it
> setup so that I can run jobs remotely from a computer on the same network
> that is not a part of the cluster. I've got a main java class that
> implements org.apache.hadoop.util.Tool and I'm able to run this job from
> the NameNode using ToolRunner.run(), setting up the JobConf, and submitting
> with JobClient.submitJob().
>
> When I try to run the same class remotely from any machine that is not the
> NameNode the job is submitted and it appears that the Map tasks
> successfully complete, but I get the following exception for all of the
> reduce tasks:
>
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/map_0.out in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
> at org.apache.hadoop.mapred.MapOutputFile.getInputFile(MapOutputFile.java:161)
> at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:220)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:398)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>
> I'm not sure how to interpret that error message. I think I'm missing some config files that are not present on the remote machine but I can't pin down exactly what I need. Does anyone have any guidance on what the above error means or how to submit jobs remotely?
>
> Thanks,
>
> --Oren
>
>
--
Regards,
..... Sudhakara.st
Re: Exception during reduce phase when running jobs remotely
Posted by sudhakara st <su...@gmail.com>.
Hello,
Check you hadoop.tmp.dir and mapred.local.dir configuration and permissions.
On Sat, Apr 27, 2013 at 12:20 AM, Oren Bumgarner <or...@gmail.com> wrote:
> I have a small hadoop cluster running 1.0.4 and I'm trying to have it
> setup so that I can run jobs remotely from a computer on the same network
> that is not a part of the cluster. I've got a main java class that
> implements org.apache.hadoop.util.Tool and I'm able to run this job from
> the NameNode using ToolRunner.run(), setting up the JobConf, and submitting
> with JobClient.submitJob().
>
> When I try to run the same class remotely from any machine that is not the
> NameNode the job is submitted and it appears that the Map tasks
> successfully complete, but I get the following exception for all of the
> reduce tasks:
>
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/map_0.out in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
> at org.apache.hadoop.mapred.MapOutputFile.getInputFile(MapOutputFile.java:161)
> at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:220)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:398)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>
> I'm not sure how to interpret that error message. I think I'm missing some config files that are not present on the remote machine but I can't pin down exactly what I need. Does anyone have any guidance on what the above error means or how to submit jobs remotely?
>
> Thanks,
>
> --Oren
>
>
--
Regards,
..... Sudhakara.st
Re: Exception during reduce phase when running jobs remotely
Posted by sudhakara st <su...@gmail.com>.
Hello,
Check you hadoop.tmp.dir and mapred.local.dir configuration and permissions.
On Sat, Apr 27, 2013 at 12:20 AM, Oren Bumgarner <or...@gmail.com> wrote:
> I have a small hadoop cluster running 1.0.4 and I'm trying to have it
> setup so that I can run jobs remotely from a computer on the same network
> that is not a part of the cluster. I've got a main java class that
> implements org.apache.hadoop.util.Tool and I'm able to run this job from
> the NameNode using ToolRunner.run(), setting up the JobConf, and submitting
> with JobClient.submitJob().
>
> When I try to run the same class remotely from any machine that is not the
> NameNode the job is submitted and it appears that the Map tasks
> successfully complete, but I get the following exception for all of the
> reduce tasks:
>
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/map_0.out in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
> at org.apache.hadoop.mapred.MapOutputFile.getInputFile(MapOutputFile.java:161)
> at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:220)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:398)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>
> I'm not sure how to interpret that error message. I think I'm missing some config files that are not present on the remote machine but I can't pin down exactly what I need. Does anyone have any guidance on what the above error means or how to submit jobs remotely?
>
> Thanks,
>
> --Oren
>
>
--
Regards,
..... Sudhakara.st
Re: Exception during reduce phase when running jobs remotely
Posted by sudhakara st <su...@gmail.com>.
Hello,
Check you hadoop.tmp.dir and mapred.local.dir configuration and permissions.
On Sat, Apr 27, 2013 at 12:20 AM, Oren Bumgarner <or...@gmail.com> wrote:
> I have a small hadoop cluster running 1.0.4 and I'm trying to have it
> setup so that I can run jobs remotely from a computer on the same network
> that is not a part of the cluster. I've got a main java class that
> implements org.apache.hadoop.util.Tool and I'm able to run this job from
> the NameNode using ToolRunner.run(), setting up the JobConf, and submitting
> with JobClient.submitJob().
>
> When I try to run the same class remotely from any machine that is not the
> NameNode the job is submitted and it appears that the Map tasks
> successfully complete, but I get the following exception for all of the
> reduce tasks:
>
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/map_0.out in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
> at org.apache.hadoop.mapred.MapOutputFile.getInputFile(MapOutputFile.java:161)
> at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:220)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:398)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>
> I'm not sure how to interpret that error message. I think I'm missing some config files that are not present on the remote machine but I can't pin down exactly what I need. Does anyone have any guidance on what the above error means or how to submit jobs remotely?
>
> Thanks,
>
> --Oren
>
>
--
Regards,
..... Sudhakara.st