You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by oleksiy <ga...@mail.ru> on 2011/10/05 12:02:14 UTC

Run hadoop Map/Reduce app from another machine

Hello,

I'm trying to find a way how to run hadoop MapReduce app from another
machine.
For instance I have *.jar file with MapReduce app it works ok when I run it
from command line for instance using this command: "hadoop jar
/usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input
/usr/joe/wordcount/output"
But in situation when I have another server (simple web app) where user can
upload jar file, specify configuration for the MapReduce app and so on. And
this server should interact with hadoop server. I mean somehow to upload
this jar file to the hadoop server and run it with attributes.

So, right now I see only one way of how to do this, it's upload jar file to
the hadoop server and run remotely command: "hadoop jar
/usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input
/usr/joe/wordcount/output". 

So may be hadoop has spatial API for doing this kind of tasks remotely? 
-- 
View this message in context: http://old.nabble.com/Run-hadoop-Map-Reduce-app-from-another-machine-tp32595264p32595264.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: Run hadoop Map/Reduce app from another machine

Posted by Yang Xiaoliang <ya...@gmail.com>.
Install hadoop on your local machine, copy the configuration files from the
remote
hadoop culuster server to your local machine(including the hosts file), then
you can
just submit a *.jar locally as before.


2011/10/5 oleksiy <ga...@mail.ru>

>
> Hello,
>
> I'm trying to find a way how to run hadoop MapReduce app from another
> machine.
> For instance I have *.jar file with MapReduce app it works ok when I run it
> from command line for instance using this command: "hadoop jar
> /usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input
> /usr/joe/wordcount/output"
> But in situation when I have another server (simple web app) where user can
> upload jar file, specify configuration for the MapReduce app and so on. And
> this server should interact with hadoop server. I mean somehow to upload
> this jar file to the hadoop server and run it with attributes.
>
> So, right now I see only one way of how to do this, it's upload jar file to
> the hadoop server and run remotely command: "hadoop jar
> /usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input
> /usr/joe/wordcount/output".
>
> So may be hadoop has spatial API for doing this kind of tasks remotely?
> --
> View this message in context:
> http://old.nabble.com/Run-hadoop-Map-Reduce-app-from-another-machine-tp32595264p32595264.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>

RE: Run hadoop Map/Reduce app from another machine

Posted by Devaraj K <de...@huawei.com>.
Hi Oleksiy,

	No need to copy the jar file and execute the hadoop command remotely
to submit job.

Hadoop provides JobClient API to submit and monitor the jobs from remote
systems/remote application. 

http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/mapred/Jo
bClient.html

Here is an example on how to use JobClient to run the job  :


     // Create a new JobConf
     JobConf job = new JobConf(new Configuration(), MyJob.class);
     
     // Specify various job-specific parameters     
     job.setJobName("myjob");
     
     job.setInputPath(new Path("in"));
     job.setOutputPath(new Path("out"));
     
     job.setMapperClass(MyJob.MyMapper.class);
     job.setReducerClass(MyJob.MyReducer.class);

     // Submit the job, then poll for progress until the job is complete
     JobClient.runJob(job);

I hope this helps to solve problem.

Devaraj K 

-----Original Message-----
From: oleksiy [mailto:gayduk.a.s.ua@mail.ru] 
Sent: Wednesday, October 05, 2011 3:32 PM
To: core-user@hadoop.apache.org
Subject: Run hadoop Map/Reduce app from another machine


Hello,

I'm trying to find a way how to run hadoop MapReduce app from another
machine.
For instance I have *.jar file with MapReduce app it works ok when I run it
from command line for instance using this command: "hadoop jar
/usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input
/usr/joe/wordcount/output"
But in situation when I have another server (simple web app) where user can
upload jar file, specify configuration for the MapReduce app and so on. And
this server should interact with hadoop server. I mean somehow to upload
this jar file to the hadoop server and run it with attributes.

So, right now I see only one way of how to do this, it's upload jar file to
the hadoop server and run remotely command: "hadoop jar
/usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input
/usr/joe/wordcount/output". 

So may be hadoop has spatial API for doing this kind of tasks remotely? 
-- 
View this message in context:
http://old.nabble.com/Run-hadoop-Map-Reduce-app-from-another-machine-tp32595
264p32595264.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.