You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2011/08/19 13:08:27 UTC

[jira] [Commented] (MAPREDUCE-2131) Aggregate JobCounters and TaskCounters in a background thread

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087647#comment-13087647 ] 

Steve Loughran commented on MAPREDUCE-2131:
-------------------------------------------

I can see some use cases where eventual consistency is OK: you don't need to force an update of everything, just a "recent" value, with some data on when that value was last collected. Decoupling aggregation from querying would achieve that, though the Counters class would need to be extended with a timestamp field, and we may want the option to force aggregation; getCounters(boolean update)

> Aggregate JobCounters and TaskCounters in a background thread
> -------------------------------------------------------------
>
>                 Key: MAPREDUCE-2131
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2131
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>
> JobTracker.getJobCounters() aggregates the counters when getting requested.
> It may consume lots of CPU if the request is too often.
> It may be good if we aggregate them in a background thread.
> Thoughts?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira