How to use Hadoop API to submit job?

[Wu Wei <we...@alibaba-inc.com>]

Hi,

I used to submit Hadoop job with the utility RunJar.main() on hadoop 
0.18. On hadoop 0.19, because the commandLineConfig of JobClient was 
null, I got a NullPointerException error when RunJar.main() calls 
GenericOptionsParser to get libJars (0.18 didn't do this call). I also 
tried the class JobShell to submit job, but it catches all exceptions 
and sends to stderr so that I cann't handle the exceptions myself.

I noticed that if I can call JobClient's setCommandLineConfig method, 
everything goes easy. But this method has default package accessibility, 
I cann't see the method out of package org.apache.hadoop.mapred.

Any advices on using Java APIs to submit job?

Wei

Re: How to use Hadoop API to submit job?

[Steve Loughran <st...@apache.org>]

Wu Wei wrote:
> Hi,
> 
> I used to submit Hadoop job with the utility RunJar.main() on hadoop 
> 0.18. On hadoop 0.19, because the commandLineConfig of JobClient was 
> null, I got a NullPointerException error when RunJar.main() calls 
> GenericOptionsParser to get libJars (0.18 didn't do this call). I also 
> tried the class JobShell to submit job, but it catches all exceptions 
> and sends to stderr so that I cann't handle the exceptions myself.
> 
> I noticed that if I can call JobClient's setCommandLineConfig method, 
> everything goes easy. But this method has default package accessibility, 
> I cann't see the method out of package org.apache.hadoop.mapred.
> 
> Any advices on using Java APIs to submit job?
> 
> Wei

Looking at my code, the line that does the work is

JobClient jc = new JobClient(jobConf);
runningJob = jc.submitJob(jobConf);

My full (LGPL) code is here : http://tinyurl.com/djk6vj

there's more work with validating input and output directories, pulling 
back the results, handling timeouts if the job doesnt complete, etc,etc, 
but that's feature creep

Re: How to use Hadoop API to submit job?

[Wu Wei <we...@alibaba-inc.com>]

My application implements Tool interface and I can submit the job using 
shell script "hadoop jar xxx.jar app". But here I don't want to use this 
script. Instead, I want to catch errors in my Java code and do some 
further proccessings.

-Wei

Amareshwari Sriramadasu wrote:
> You should implement Tool interface and submit jobs.
> For example see org.apache.hadoop.examples.WordCount
>
> -Amareshwari
> Wu Wei wrote:
>> Hi,
>>
>> I used to submit Hadoop job with the utility RunJar.main() on hadoop 
>> 0.18. On hadoop 0.19, because the commandLineConfig of JobClient was 
>> null, I got a NullPointerException error when RunJar.main() calls 
>> GenericOptionsParser to get libJars (0.18 didn't do this call). I 
>> also tried the class JobShell to submit job, but it catches all 
>> exceptions and sends to stderr so that I cann't handle the exceptions 
>> myself.
>>
>> I noticed that if I can call JobClient's setCommandLineConfig method, 
>> everything goes easy. But this method has default package 
>> accessibility, I cann't see the method out of package 
>> org.apache.hadoop.mapred.
>>
>> Any advices on using Java APIs to submit job?
>>
>> Wei
>
>
>

Re: How to use Hadoop API to submit job?

[Amareshwari Sriramadasu <am...@yahoo-inc.com>]

You should implement Tool interface and submit jobs.
For example see org.apache.hadoop.examples.WordCount

-Amareshwari
Wu Wei wrote:
> Hi,
>
> I used to submit Hadoop job with the utility RunJar.main() on hadoop 
> 0.18. On hadoop 0.19, because the commandLineConfig of JobClient was 
> null, I got a NullPointerException error when RunJar.main() calls 
> GenericOptionsParser to get libJars (0.18 didn't do this call). I also 
> tried the class JobShell to submit job, but it catches all exceptions 
> and sends to stderr so that I cann't handle the exceptions myself.
>
> I noticed that if I can call JobClient's setCommandLineConfig method, 
> everything goes easy. But this method has default package 
> accessibility, I cann't see the method out of package 
> org.apache.hadoop.mapred.
>
> Any advices on using Java APIs to submit job?
>
> Wei