You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Some Body <so...@squareplanet.de> on 2010/06/21 15:30:37 UTC
new/old APIs
I still haven't figured out how to query the JobTracker for a specific running job using the new API.
Is this at all possible? The old/new APIs are driving me crazy.
As far as I understand the old API entailed using mapred.JobClient & JobConf
JobConf jobConf = new JobConf(new Configuration(), MyJob.class);
....
RunningJob runningJob = JobClient.submitJob(jobConf);
But in the new API I'm using mapreduce.Job like this:
130: Job job = new Job(new Configuration(), "MyJobName");
131: job.submit();
132: // The job starts ok here but the job.getJobID() gives me a null pointer.
133: System.err.printf("submitted job %s\n", job.getJobID().toString());
After submitting the job with job.submit() above I can do this to get all jobs,
100: JobConf jobConf = new JobConf(config, MyDriver.class);
101: JobClient jc = new JobClient(jobConf);
102: JobStatus[] allJobs = jc.getAllJobs();
103: for (int i = 0; i<allJobs.length; i++) {
104: // check running jobs
105: int status = allJobs[i].getRunState();
106: if ( status == 1 ) {
107: JobID jid = allJobs[i].getJobID();
108: // compare this id or name to the id/name from my lock file
109: System.err.printf("job %s is still running\n", jid.toString());
110: }
111: }
but I still need a way to retrieve the job's ID or name after line 131 above,
because I'd be writing this ID (or name) to a lock file and then killing the
job in the loop before starting a new job.
Thanks,
Alan
Re: new/old APIs
Posted by Ted Yu <yu...@gmail.com>.
Minor correction:
job Id is in the tracking URL
:50030/jobdetails.jsp?jobid=job_201006171925_0106&refresh=0
On Mon, Jun 21, 2010 at 9:36 AM, Ted Yu <yu...@gmail.com> wrote:
> In new API, info is a private member of Job. I haven't found an easy way to
> retrieve what you wanted.
>
> You can call getTrackingURL() and try parsing the HTML from the contents of
> HTML. :-)
>
> You can also file a JIRA about this.
>
> FYI mapred is going to be undeprecated. I would suggest spending your time
> on other part of your code.
>
>
>
> On Mon, Jun 21, 2010 at 6:30 AM, Some Body <so...@squareplanet.de>wrote:
>
>> I still haven't figured out how to query the JobTracker for a specific
>> running job using the new API.
>> Is this at all possible? The old/new APIs are driving me crazy.
>>
>> As far as I understand the old API entailed using mapred.JobClient &
>> JobConf
>>
>> JobConf jobConf = new JobConf(new Configuration(), MyJob.class);
>> ....
>> RunningJob runningJob = JobClient.submitJob(jobConf);
>>
>> But in the new API I'm using mapreduce.Job like this:
>>
>> 130: Job job = new Job(new Configuration(), "MyJobName");
>> 131: job.submit();
>> 132: // The job starts ok here but the job.getJobID() gives me a null
>> pointer.
>> 133: System.err.printf("submitted job %s\n", job.getJobID().toString());
>>
>> After submitting the job with job.submit() above I can do this to get all
>> jobs,
>>
>> 100: JobConf jobConf = new JobConf(config, MyDriver.class);
>> 101: JobClient jc = new JobClient(jobConf);
>> 102: JobStatus[] allJobs = jc.getAllJobs();
>> 103: for (int i = 0; i<allJobs.length; i++) {
>> 104: // check running jobs
>> 105: int status = allJobs[i].getRunState();
>> 106: if ( status == 1 ) {
>> 107: JobID jid = allJobs[i].getJobID();
>> 108: // compare this id or name to the id/name from my lock file
>> 109: System.err.printf("job %s is still running\n",
>> jid.toString());
>> 110: }
>> 111: }
>>
>> but I still need a way to retrieve the job's ID or name after line 131
>> above,
>> because I'd be writing this ID (or name) to a lock file and then killing
>> the
>> job in the loop before starting a new job.
>>
>> Thanks,
>> Alan
>>
>
>
Re: new/old APIs
Posted by Ted Yu <yu...@gmail.com>.
In new API, info is a private member of Job. I haven't found an easy way to
retrieve what you wanted.
You can call getTrackingURL() and try parsing the HTML from the contents of
HTML. :-)
You can also file a JIRA about this.
FYI mapred is going to be undeprecated. I would suggest spending your time
on other part of your code.
On Mon, Jun 21, 2010 at 6:30 AM, Some Body <so...@squareplanet.de> wrote:
> I still haven't figured out how to query the JobTracker for a specific
> running job using the new API.
> Is this at all possible? The old/new APIs are driving me crazy.
>
> As far as I understand the old API entailed using mapred.JobClient &
> JobConf
>
> JobConf jobConf = new JobConf(new Configuration(), MyJob.class);
> ....
> RunningJob runningJob = JobClient.submitJob(jobConf);
>
> But in the new API I'm using mapreduce.Job like this:
>
> 130: Job job = new Job(new Configuration(), "MyJobName");
> 131: job.submit();
> 132: // The job starts ok here but the job.getJobID() gives me a null
> pointer.
> 133: System.err.printf("submitted job %s\n", job.getJobID().toString());
>
> After submitting the job with job.submit() above I can do this to get all
> jobs,
>
> 100: JobConf jobConf = new JobConf(config, MyDriver.class);
> 101: JobClient jc = new JobClient(jobConf);
> 102: JobStatus[] allJobs = jc.getAllJobs();
> 103: for (int i = 0; i<allJobs.length; i++) {
> 104: // check running jobs
> 105: int status = allJobs[i].getRunState();
> 106: if ( status == 1 ) {
> 107: JobID jid = allJobs[i].getJobID();
> 108: // compare this id or name to the id/name from my lock file
> 109: System.err.printf("job %s is still running\n", jid.toString());
> 110: }
> 111: }
>
> but I still need a way to retrieve the job's ID or name after line 131
> above,
> because I'd be writing this ID (or name) to a lock file and then killing
> the
> job in the loop before starting a new job.
>
> Thanks,
> Alan
>