Question about reporting progress in mapper tasks. 0.15.3

[Jason Venner <ja...@attributor.com>]

I have a mapper that for each task does extensive computation. In the 
computation, I increment a counter once per major operation (about once 
every 5 seconds). I can see this happening by the log messages, that 
happen around the reporter.incrCounter call.

Still my mapper is getting killed after not reporting for 60X seconds, 
it is clear that that incrementing a counter is insufficient disable the 
kill timeout.

How do you disable the kill timeout?



-- 
Jason Venner
Attributor - Publish with Confidence <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers, contact if interested

RE: Question about reporting progress in mapper tasks. 0.15.3 - solved

[Joydeep Sen Sarma <js...@facebook.com>]

Well .. counters are back in 16 as progress indicators (I think):

http://issues.apache.org/jira/browse/HADOOP-2189

-----Original Message-----
From: Jason Venner [mailto:jason@attributor.com] 
Sent: Tuesday, April 15, 2008 6:53 PM
To: core-user@hadoop.apache.org
Subject: Re: Question about reporting progress in mapper tasks. 0.15.3 -
solved

Well, on deeper reading of the code and the documentation, 
reporter.progress(), is the required call.

Jason Venner wrote:
> I have a mapper that for each task does extensive computation. In the 
> computation, I increment a counter once per major operation (about 
> once every 5 seconds). I can see this happening by the log messages, 
> that happen around the reporter.incrCounter call.
>
> Still my mapper is getting killed after not reporting for 60X seconds,

> it is clear that that incrementing a counter is insufficient disable 
> the kill timeout.
>
> How do you disable the kill timeout?
>
>
>

Re: Question about reporting progress in mapper tasks. 0.15.3 - solved

[Jason Venner <ja...@attributor.com>]

Well, on deeper reading of the code and the documentation, 
reporter.progress(), is the required call.

Jason Venner wrote:
> I have a mapper that for each task does extensive computation. In the 
> computation, I increment a counter once per major operation (about 
> once every 5 seconds). I can see this happening by the log messages, 
> that happen around the reporter.incrCounter call.
>
> Still my mapper is getting killed after not reporting for 60X seconds, 
> it is clear that that incrementing a counter is insufficient disable 
> the kill timeout.
>
> How do you disable the kill timeout?
>
>
>