Getting job progress in java application

[Ondřej Klimpera <kl...@fit.cvut.cz>]

Hello I'd like to ask you what is the preferred way of getting running 
jobs progress from Java application, that has executed them.

Im using Hadoop 0.20.203, tried job.end.notification.url property that 
works well, but as the property name says, it sends only job end 
notifications.

What I need is to get updates on map() and reduce() progress.

Please help how to do this.

Thanks.
Ondrej Klimpera

Re: Getting job progress in java application

[Harsh J <ha...@cloudera.com>]

Ondřej,

On Mon, Apr 30, 2012 at 2:29 PM, Ondřej Klimpera <kl...@fit.cvut.cz> wrote:
> Thanks a lot, checked the Docs and submitJob() method did the job.
>
> Two more question please:)
>
> [1] My app is running on Hadoop 0.20.203, if I upgrade the libraries to
> 1.0.X, will the old API work, or it is necessary to rewrite map() and
> reduce() functions to new API?

The MR API hasn't changed between 0.20.203 and 1.0.2, so yes you can
use existing jars without needing to recompile.

> [2] Does the new API support MultipleOutputs?

There is a version of New API MultipleOutputs backported in 1.0.1+ you may use.

-- 
Harsh J

Re: Getting job progress in java application

[Ondřej Klimpera <kl...@fit.cvut.cz>]

Thanks a lot, checked the Docs and submitJob() method did the job.

Two more question please:)

[1] My app is running on Hadoop 0.20.203, if I upgrade the libraries to 
1.0.X, will the old API work, or it is necessary to rewrite map() and 
reduce() functions to new API?

[2] Does the new API support MultipleOutputs?

Thanks again.



On 04/30/2012 12:32 AM, Bill Graham wrote:
> Take a look at the JobClient API. You can use that to get the current
> progress of a running job.
>
> On Sunday, April 29, 2012, Ondřej Klimpera wrote:
>
>> Hello I'd like to ask you what is the preferred way of getting running
>> jobs progress from Java application, that has executed them.
>>
>> Im using Hadoop 0.20.203, tried job.end.notification.url property that
>> works well, but as the property name says, it sends only job end
>> notifications.
>>
>> What I need is to get updates on map() and reduce() progress.
>>
>> Please help how to do this.
>>
>> Thanks.
>> Ondrej Klimpera
>>
>>

Re: Getting job progress in java application

[Bill Graham <bi...@gmail.com>]

Take a look at the JobClient API. You can use that to get the current
progress of a running job.

On Sunday, April 29, 2012, Ondřej Klimpera wrote:

> Hello I'd like to ask you what is the preferred way of getting running
> jobs progress from Java application, that has executed them.
>
> Im using Hadoop 0.20.203, tried job.end.notification.url property that
> works well, but as the property name says, it sends only job end
> notifications.
>
> What I need is to get updates on map() and reduce() progress.
>
> Please help how to do this.
>
> Thanks.
> Ondrej Klimpera
>
>

-- 
*Note that I'm no longer using my Yahoo! email address. Please email me at
billgraham@gmail.com going forward.*