Getting (or setting) a job ID

[Adam Phelps <am...@opendns.com>]

We have intermittently seen cases where a job will "freeze" for some as 
yet unknown reason, and thereby block other processes waiting for that 
job to complete.  I'm trying to modify our job-launching scripts to kill 
the job if it doesn't complete in a reasonable amount of time, however 
to do that I would need to know the job ID (ie along the lines of 
job_201104292017_9008).

As far as I can tell there is no good way to map the name we use to 
create the job to the job ID from the command line (none of the "hadoop 
job" options do it).  Am I missing something in the CLI that would help 
with this?  Alternate ideas are also welcome.

- Adam

Re: Getting (or setting) a job ID

[Harsh J <ha...@cloudera.com>]

Hey Adam,

On Wed, May 11, 2011 at 5:17 AM, Adam Phelps <am...@opendns.com> wrote:
> We could make a change of that nature.  We currently use
> job.waitForCompletion() to submit jobs, if we switch to job.submit() is
> there an alternate blocking call to wait for the job to finish other than
> polling job.isComplete()?  I'm not currently seeing anything in the API.

Polling isComplete() is exactly what the waitForCompletion does
underneath anyway. It is fine to use the same.

This is what the waitForCompletion entails:
while (!isComplete()) {
        try {
          Thread.sleep(5000-(default)-millis);
        } catch (InterruptedException ie) {
        }
}

-- 
Harsh J

Re: Getting (or setting) a job ID

[Adam Phelps <am...@opendns.com>]

We could make a change of that nature.  We currently use 
job.waitForCompletion() to submit jobs, if we switch to job.submit() is 
there an alternate blocking call to wait for the job to finish other 
than polling job.isComplete()?  I'm not currently seeing anything in the 
API.

- Adam

On 5/10/11 3:47 PM, Aaron Baff wrote:
> As part of the job submission, once it's submitted, grab the JobID from that object and print it out on STDOUT or to a file and have your startup script(s) parse it out from there.
>
> --Aaron
>
> -----Original Message-----
> From: Adam Phelps [mailto:amp@opendns.com]
> Sent: Tuesday, May 10, 2011 3:45 PM
> To: mapreduce-user@hadoop.apache.org
> Subject: Getting (or setting) a job ID
>
> We have intermittently seen cases where a job will "freeze" for some as
> yet unknown reason, and thereby block other processes waiting for that
> job to complete.  I'm trying to modify our job-launching scripts to kill
> the job if it doesn't complete in a reasonable amount of time, however
> to do that I would need to know the job ID (ie along the lines of
> job_201104292017_9008).
>
> As far as I can tell there is no good way to map the name we use to
> create the job to the job ID from the command line (none of the "hadoop
> job" options do it).  Am I missing something in the CLI that would help
> with this?  Alternate ideas are also welcome.
>
> - Adam
>

RE: Getting (or setting) a job ID

[Aaron Baff <Aa...@telescope.tv>]

As part of the job submission, once it's submitted, grab the JobID from that object and print it out on STDOUT or to a file and have your startup script(s) parse it out from there.

--Aaron

-----Original Message-----
From: Adam Phelps [mailto:amp@opendns.com]
Sent: Tuesday, May 10, 2011 3:45 PM
To: mapreduce-user@hadoop.apache.org
Subject: Getting (or setting) a job ID

We have intermittently seen cases where a job will "freeze" for some as
yet unknown reason, and thereby block other processes waiting for that
job to complete.  I'm trying to modify our job-launching scripts to kill
the job if it doesn't complete in a reasonable amount of time, however
to do that I would need to know the job ID (ie along the lines of
job_201104292017_9008).

As far as I can tell there is no good way to map the name we use to
create the job to the job ID from the command line (none of the "hadoop
job" options do it).  Am I missing something in the CLI that would help
with this?  Alternate ideas are also welcome.

- Adam