Is it possible to submit job to JobClient and exit immediately?

[Andrew <gu...@nigma.ru>]

For now, I use such code blocks in all my MR jobs:
	
try {
	JobClient.runJob(job);
} catch (IOException exc) {
	LOG.info("Job failed", exc);
}
System.exit(0);

But this code waits until MR job to complete. Thus, I have to run it on 
machine that is always online to jobtracker.

My purpose is to write code that submits job to job tracker and exits without 
waiting for job to complete. I've tried 

try {
	JobID jobid = new JobClient(job).submitJob(job).getID();
} catch (IOException exc) {
	LOG.info("Job failed", exc);
}
System.exit(0);

This code fails  with "java.lang.IllegalStateException: Shutdown in progress". 
It seems, that JobClient creates non-daemon threads on submitJob() 
invocation.


So, is there the way to submit job and exit immediately?

-- 
Andrew Gudkov
PGP key id: CB9F07D8 (cryptonomicon.mit.edu)
Jabber: gudok@jabber.ru

Re: Is it possible to submit job to JobClient and exit immediately?

[Andrew <gu...@nigma.ru>]

> Can you check what state the jobtracker via its web-ui? Can you see what
> is happening to the jobtracker by checking its logs?
I figured it out: submit waits for completion only on "local" tasktracker. A 
standalone thread is created for LocalJobRunner in this case.

Everything is ok on real cluster.


-- 
Andrew Gudkov
PGP key id: CB9F07D8 (cryptonomicon.mit.edu)
Jabber: gudok@jabber.ru

Re: Is it possible to submit job to JobClient and exit immediately?

[Amar Kamat <am...@yahoo-inc.com>]

Andrew wrote:
> For now, I use such code blocks in all my MR jobs:
> 	
> try {
> 	JobClient.runJob(job);
>   
JobClient jc = new JobClient(job);
jc.submitJob(job); // submits a job and comes out
> } catch (IOException exc) {
> 	LOG.info("Job failed", exc);
> }
> System.exit(0);
>
> But this code waits until MR job to complete. Thus, I have to run it on 
> machine that is always online to jobtracker.
>
> My purpose is to write code that submits job to job tracker and exits without 
> waiting for job to complete. I've tried 
>
> try {
> 	JobID jobid = new JobClient(job).submitJob(job).getID();
> } catch (IOException exc) {
> 	LOG.info("Job failed", exc);
> }
> System.exit(0);
>
> This code fails  with "java.lang.IllegalStateException: Shutdown in progress". 
>   
Can you check what state the jobtracker via its web-ui? Can you see what 
is happening to the jobtracker by checking its logs?
Amar
> It seems, that JobClient creates non-daemon threads on submitJob() 
> invocation.
>
>
> So, is there the way to submit job and exit immediately?
>
>