Re: how does hadoop handle the counter of the failed task and speculative task

[Arun C Murthy <ac...@yahoo-inc.com>]

On Dec 25, 2010, at 10:08 PM, lei liu wrote:

> I define the counter to count the bad records, there is below code  
> in map task;
>  reporter.incrCounter("bad',"
> records', 1),
>
> When the job is completed, the pritnt the result to use below code:
> long total = counters.findCounter("bad","records").getCounter();
>
>
> But I have two questions about the counter:
> 1. If the map task is retried 4 times, and the last map task is  
> successful,  I think the counter of the other three map tasks should  
> be not included in ultima result, is that right?
> 2. If there are speculative tasks,  I think the counter of  
> speculative tasks should be not included in ultima result, is that  
> right?
>

Yes to both. Counters are kept only from the 'successful' task- 
attempt, the others (failed, speculative etc.) are discarded.

Arun