You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Brian Bloniarz (Commented) (JIRA)" <ji...@apache.org> on 2011/11/28 22:19:40 UTC
[jira] [Commented] (MAPREDUCE-2835) Make per-job counter limits
configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158786#comment-13158786 ]
Brian Bloniarz commented on MAPREDUCE-2835:
-------------------------------------------
If MAPREDUCE-1943 was introduced to avoid memory corruption, 50 groups
and 120 counters is an awfully low limit. The submitter complained of
jobs which had millions of counters. I believe that the current limits
will cap Counters after consuming around 32KB of memory, which IMHO is
not a reasonable tradeoff in footprint vs inconvenience.
Regarding inconvenience: note that:
(1) The counter & group limits are currently hard fails. Jobs will stop working.
(2) it's a change to the current behavior; jobs which worked under 0.20 will now fail.
If the limits were raised to (for example):
10000 counters
1000 groups
1000 char name limit
they would prevent crashing all but the most comically limited clusters
(I measure Counters consuming 21mb with these limits), and trip up far fewer
programmers. For the same reason, I think Tom's patch should be applied so they're
expandable without requiring a recompilation of hadoop.
> Make per-job counter limits configurable
> ----------------------------------------
>
> Key: MAPREDUCE-2835
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2835
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.20.204.0
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-2835.patch, MAPREDUCE-2835.patch
>
>
> The per-job counter limits introduced in MAPREDUCE-1943 are fixed, except for the total number allowed per job (mapreduce.job.counters.limit). It would be useful to make them all configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira