Run job not from namenode

[Andrey Pankov <ap...@iponweb.net>]

Hi all,

Currently I'm able to run map-reduce jobs from box where NameNode and 
JobTracker are running. But I'd like to run my jobs from separate box, 
from which I have access to HDFS. I have updated params fs.default.name 
and  mapred.job.tracker in local hadoop dir to point to the clusters 
master. Now Hadoop returns me following error:

username@ip-10-251-96-129:/usr/local/hadoop-0.16.0$ bin/hadoop jar 
hadoop-0.16.0-examples.jar wordcount /user/username/gutenberg 
/user/username/gutenberg-output
08/03/31 10:21:46 INFO mapred.FileInputFormat: Total input paths to 
process : 3
org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
/mnt/hadoop/mapred/system/job_200803210640_0852/job.xml: No such file or 
directory
         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:159)
         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:133)
         at 
org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1083)
  ...

Here account 'username' has passwordless access to master box. Cluster 
runs over EC2.

As a variant I can run tasks via ssh, i.e.

ssh master /usr/local/hadoop-0.16.0bin/hadoop jar 
/home/username/jobs/hadoop-0.16.0-examples.jar wordcount 
/user/username/gutenberg /user/username/gutenberg-output

But you need to put your jar file to the NameNode box before you run it.

Thanks in advance.

-- 
Andrey Pankov

Re: Run job not from namenode

[Amar Kamat <am...@yahoo-inc.com>]

Andrey Pankov wrote:
> Hi all,
>
> Currently I'm able to run map-reduce jobs from box where NameNode and 
> JobTracker are running. But I'd like to run my jobs from separate box, 
> from which I have access to HDFS. I have updated params 
> fs.default.name and  mapred.job.tracker in local hadoop dir to point 
> to the clusters master. Now Hadoop returns me following error:
>
> username@ip-10-251-96-129:/usr/local/hadoop-0.16.0$ bin/hadoop jar 
> hadoop-0.16.0-examples.jar wordcount /user/username/gutenberg 
> /user/username/gutenberg-output
> 08/03/31 10:21:46 INFO mapred.FileInputFormat: Total input paths to 
> process : 3
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
> /mnt/hadoop/mapred/system/job_200803210640_0852/job.xml: No such file 
> or directory
I tried running this with trunk and it works. Started a cluster and 
fired a job from my local machine with the local hadoop-site.xml 
pointing to the JobTracker and Namenode. Can you provide us with more 
details as to how to reproduce the problem?
Amar
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:159)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:133)
>         at 
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1083)
>  ...
>
> Here account 'username' has passwordless access to master box. Cluster 
> runs over EC2.
>
> As a variant I can run tasks via ssh, i.e.
>
> ssh master /usr/local/hadoop-0.16.0bin/hadoop jar 
> /home/username/jobs/hadoop-0.16.0-examples.jar wordcount 
> /user/username/gutenberg /user/username/gutenberg-output
>
> But you need to put your jar file to the NameNode box before you run it.
>
> Thanks in advance.
>