hadoop job controller

[Elia Mazzawi <el...@casalemedia.com>]

I'm writing a perl program to submit jobs to the cluster,
then wait for the jobs to finish, and check that they have completed 
successfully.

I have some questions,

this shows what is running
./hadoop job  -list

and this shows the completion
./hadoop job -status  job_200903061521_0045


but i want something that just says pass / fail
cause with these, i have to check that its done then check that its 100% 
completed.

which must exist since the webapp jobtracker.jsp knows what is what.

also a controller like that must have been written many times already,  
are there any around?

Regards,
Elia

Re: job status from command prompt

[Amareshwari Sriramadasu <am...@yahoo-inc.com>]

Elia Mazzawi wrote:
> is there a command that i can run from the shell that says this job 
> passed / failed
>
> I found these but they don't really say pass/fail they only say what 
> is running and percent complete.
>
> this shows what is running
> ./hadoop job -list
>
> and this shows the completion
> ./hadoop job -status job_200903061521_0045
The following command lists all jobs in prep, running, completed:
./hadoop job -list all

-Amareshwari

job status from command prompt

[Elia Mazzawi <el...@casalemedia.com>]

is there a command that i can run from the shell that says this job 
passed / failed

I found these but they don't really say pass/fail they only say what is 
running and percent complete.

this shows what is running
./hadoop job -list

and this shows the completion
./hadoop job -status job_200903061521_0045

Re: hadoop job controller

[Stefan Podkowinski <sp...@gmail.com>]

You can get the job progress and completion status through an instance
of org.apache.hadoop.mapred.JobClient . If you really want to use perl
I guess you still need to write a small java application that talks to
perl and JobClient on the other side.
Theres also some support for Thrift in the hadoop contrib package, but
I'm not sure if it exposes any job client related methods.

On Thu, Apr 2, 2009 at 12:46 AM, Elia Mazzawi
<el...@casalemedia.com> wrote:
>
> I'm writing a perl program to submit jobs to the cluster,
> then wait for the jobs to finish, and check that they have completed
> successfully.
>
> I have some questions,
>
> this shows what is running
> ./hadoop job  -list
>
> and this shows the completion
> ./hadoop job -status  job_200903061521_0045
>
>
> but i want something that just says pass / fail
> cause with these, i have to check that its done then check that its 100%
> completed.
>
> which must exist since the webapp jobtracker.jsp knows what is what.
>
> also a controller like that must have been written many times already,  are
> there any around?
>
> Regards,
> Elia
>