You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by er...@yahoo.com on 2011/01/31 16:09:43 UTC
submitting jobs from a webapp
Hi,
Sorry if this is a trivial question but I have been battling with it for a
while. I have a question about submitting jobs from a web application
potentially deployed on different machines than HBase. I am using
springframework and implemented a simple JobService class that acts as a spring
service. I can then implement all my jobs in separate classes and pass them to
my jobService for execution. When I run jobs through the hadoop command line,
the job shows up in the job tracker. However, when I submit jobs through my
webapp, it does not. Since I can step through the execution of my job in my
eclipse environment, I don't think this job is running in hadoop/hbase's
distributed environment. Is that right?
1. In short, how can I submit a job from my webapp and view its progress on the
jobtracker?
2. Additionally, I would also like to be able to get this information to display
it in my own web application (so that users can track the status of their
running jobs). How can I query the Hadoop job tracker?
Thanks a lot for your help,
-Eric
here is snipets of my code:
class myJob extends IJob {
...
public Job createSubmittableJob(Configuration conf, String[] args)
Path inputDir = new Path(inputDir);
job = new Job(conf, jobName);
job.setJarByClass(Uploader.class);
FileInputFormat.setInputPaths(job, inputDir);
job.setInputFormatClass(TextInputFormat.class);
job.setMapperClass(TransactionLoader.class);
TableMapReduceUtil.initTableReducerJob(tableName, null, job);
job.setNumReduceTasks(0);
}
...
}
class myJobService {
...
public int submitJob(String name, IJob zjob, String[] args) {
job = zjob.createSubmittableJob(dsm.getHBaseConfiguration(), def, otherArgs);
// "Submit the job to the cluster and wait for it to finish."
return job.waitForCompletion(true) ? 0 : 1;
}
...
}
Re: submitting jobs from a webapp
Posted by Jean-Daniel Cryans <jd...@apache.org>.
(moving this to the user mailing list where it belongs)
You need to make sure that your webapp knows the address of the
JobTracker, usually this is done by either putting mapred-site.xml on
your app's classpath or you can set mapred.job.tracker correctly so
that in createSubmittableJob you would have:
conf.set("mapred.job.tracker", "whatever the address is:and the port");
Else it will run in local mode.
J-D
On Mon, Jan 31, 2011 at 7:09 AM, <er...@yahoo.com> wrote:
> Hi,
>
> Sorry if this is a trivial question but I have been battling with it for a
> while. I have a question about submitting jobs from a web application
> potentially deployed on different machines than HBase. I am using
> springframework and implemented a simple JobService class that acts as a spring
> service. I can then implement all my jobs in separate classes and pass them to
> my jobService for execution. When I run jobs through the hadoop command line,
> the job shows up in the job tracker. However, when I submit jobs through my
> webapp, it does not. Since I can step through the execution of my job in my
> eclipse environment, I don't think this job is running in hadoop/hbase's
> distributed environment. Is that right?
>
> 1. In short, how can I submit a job from my webapp and view its progress on the
> jobtracker?
>
> 2. Additionally, I would also like to be able to get this information to display
> it in my own web application (so that users can track the status of their
> running jobs). How can I query the Hadoop job tracker?
>
> Thanks a lot for your help,
> -Eric
>
> here is snipets of my code:
>
> class myJob extends IJob {
> ...
> public Job createSubmittableJob(Configuration conf, String[] args)
> Path inputDir = new Path(inputDir);
> job = new Job(conf, jobName);
> job.setJarByClass(Uploader.class);
> FileInputFormat.setInputPaths(job, inputDir);
> job.setInputFormatClass(TextInputFormat.class);
> job.setMapperClass(TransactionLoader.class);
> TableMapReduceUtil.initTableReducerJob(tableName, null, job);
> job.setNumReduceTasks(0);
> }
> ...
> }
>
> class myJobService {
> ...
>
> public int submitJob(String name, IJob zjob, String[] args) {
>
> job = zjob.createSubmittableJob(dsm.getHBaseConfiguration(), def, otherArgs);
> // "Submit the job to the cluster and wait for it to finish."
> return job.waitForCompletion(true) ? 0 : 1;
> }
> ...
> }
Re: submitting jobs from a webapp
Posted by Stack <st...@duboce.net>.
On Mon, Jan 31, 2011 at 7:09 AM, <er...@yahoo.com> wrote:
> 1. In short, how can I submit a job from my webapp and view its progress on the
> jobtracker?
>
As J-D says, you'll need to make sure your webapp knows where to make
submission -- thats being able to read configs (add a config. screen
to your webapp where user can fill in location or stick the config. in
the dir your container watches for config. changes).
You could have your webapp exec the commands you'd normally run in shell.
> 2. Additionally, I would also like to be able to get this information to display
> it in my own web application (so that users can track the status of their
> running jobs). How can I query the Hadoop job tracker?
>
You can scrape the jobtracker UI (yuck) or you can interrogate JMX to
see if this info is exposed there (I don't know).
St.Ack
Re: submitting jobs from a webapp
Posted by Jean-Daniel Cryans <jd...@apache.org>.
(moving this to the user mailing list where it belongs)
You need to make sure that your webapp knows the address of the
JobTracker, usually this is done by either putting mapred-site.xml on
your app's classpath or you can set mapred.job.tracker correctly so
that in createSubmittableJob you would have:
conf.set("mapred.job.tracker", "whatever the address is:and the port");
Else it will run in local mode.
J-D
On Mon, Jan 31, 2011 at 7:09 AM, <er...@yahoo.com> wrote:
> Hi,
>
> Sorry if this is a trivial question but I have been battling with it for a
> while. I have a question about submitting jobs from a web application
> potentially deployed on different machines than HBase. I am using
> springframework and implemented a simple JobService class that acts as a spring
> service. I can then implement all my jobs in separate classes and pass them to
> my jobService for execution. When I run jobs through the hadoop command line,
> the job shows up in the job tracker. However, when I submit jobs through my
> webapp, it does not. Since I can step through the execution of my job in my
> eclipse environment, I don't think this job is running in hadoop/hbase's
> distributed environment. Is that right?
>
> 1. In short, how can I submit a job from my webapp and view its progress on the
> jobtracker?
>
> 2. Additionally, I would also like to be able to get this information to display
> it in my own web application (so that users can track the status of their
> running jobs). How can I query the Hadoop job tracker?
>
> Thanks a lot for your help,
> -Eric
>
> here is snipets of my code:
>
> class myJob extends IJob {
> ...
> public Job createSubmittableJob(Configuration conf, String[] args)
> Path inputDir = new Path(inputDir);
> job = new Job(conf, jobName);
> job.setJarByClass(Uploader.class);
> FileInputFormat.setInputPaths(job, inputDir);
> job.setInputFormatClass(TextInputFormat.class);
> job.setMapperClass(TransactionLoader.class);
> TableMapReduceUtil.initTableReducerJob(tableName, null, job);
> job.setNumReduceTasks(0);
> }
> ...
> }
>
> class myJobService {
> ...
>
> public int submitJob(String name, IJob zjob, String[] args) {
>
> job = zjob.createSubmittableJob(dsm.getHBaseConfiguration(), def, otherArgs);
> // "Submit the job to the cluster and wait for it to finish."
> return job.waitForCompletion(true) ? 0 : 1;
> }
> ...
> }