You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Severin Corsten (Created) (JIRA)" <ji...@apache.org> on 2011/09/30 05:31:47 UTC
[jira] [Created] (GIRAPH-43) GiraphJob: Counter Limit is not set
GiraphJob: Counter Limit is not set
-----------------------------------
Key: GIRAPH-43
URL: https://issues.apache.org/jira/browse/GIRAPH-43
Project: Giraph
Issue Type: Bug
Components: graph
Affects Versions: 0.70.0
Reporter: Severin Corsten
Priority: Minor
I have got a problem regarding Counters:
For testing and benchmarking I added a lot of timers bud now the task fails with this Exception:
org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120
I don't understand why this happens. Within GiraphJob the counter limit is set to 512.
In the cluster configuration there is no value specified.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-43) GiraphJob: Counter Limit is not set
Posted by "Avery Ching (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-43?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117891#comment-13117891 ]
Avery Ching commented on GIRAPH-43:
-----------------------------------
Yeah, this kind of sucks. One thing we could do is say keep the timers (counters) only for the last n supersteps. This way we don't grow infinitely. I.e use two counters per superstep
superstep a = 3
superstep a milliseconds = 70
superstep b = 4
superstep b milliseconds = 71
...
For jobs with a lot of supersteps, we can just grow forever given this limitation.
> GiraphJob: Counter Limit is not set
> -----------------------------------
>
> Key: GIRAPH-43
> URL: https://issues.apache.org/jira/browse/GIRAPH-43
> Project: Giraph
> Issue Type: Bug
> Components: graph
> Affects Versions: 0.70.0
> Reporter: Severin Corsten
> Priority: Minor
>
> I have got a problem regarding Counters:
> For testing and benchmarking I added a lot of timers bud now the task fails with this Exception:
> org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120
> I don't understand why this happens. Within GiraphJob the counter limit is set to 512.
> In the cluster configuration there is no value specified.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (GIRAPH-43) GiraphJob: Counter Limit is not set
Posted by "Jakob Homan (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-43?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan resolved GIRAPH-43.
-------------------------------
Resolution: Not A Problem
Resolving as not a problem, since this is the intended behavior in 203. If it needs to be addressed from Giraph's side, another JIRA can handle it.
> GiraphJob: Counter Limit is not set
> -----------------------------------
>
> Key: GIRAPH-43
> URL: https://issues.apache.org/jira/browse/GIRAPH-43
> Project: Giraph
> Issue Type: Bug
> Components: graph
> Affects Versions: 0.70.0
> Reporter: Severin Corsten
> Priority: Minor
>
> I have got a problem regarding Counters:
> For testing and benchmarking I added a lot of timers bud now the task fails with this Exception:
> org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120
> I don't understand why this happens. Within GiraphJob the counter limit is set to 512.
> In the cluster configuration there is no value specified.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-43) GiraphJob: Counter Limit is not set
Posted by "Jakob Homan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-43?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117888#comment-13117888 ]
Jakob Homan commented on GIRAPH-43:
-----------------------------------
Yes, 120 is the default. It's a cluster config, so it can't be overridden on a per job basis (otherwise, everyone would just override it when inconvenient and jeopardize the JobTracker). If you're in charge of the cluster, go ahead and modify mapreduce.job.counters.limit in your conf file. If not, beg and plead with whoever is, or consider using fewer counters.
> GiraphJob: Counter Limit is not set
> -----------------------------------
>
> Key: GIRAPH-43
> URL: https://issues.apache.org/jira/browse/GIRAPH-43
> Project: Giraph
> Issue Type: Bug
> Components: graph
> Affects Versions: 0.70.0
> Reporter: Severin Corsten
> Priority: Minor
>
> I have got a problem regarding Counters:
> For testing and benchmarking I added a lot of timers bud now the task fails with this Exception:
> org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120
> I don't understand why this happens. Within GiraphJob the counter limit is set to 512.
> In the cluster configuration there is no value specified.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-43) GiraphJob: Counter Limit is not set
Posted by "Severin Corsten (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-43?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117858#comment-13117858 ]
Severin Corsten commented on GIRAPH-43:
---------------------------------------
Its 0.20.203.0.
> GiraphJob: Counter Limit is not set
> -----------------------------------
>
> Key: GIRAPH-43
> URL: https://issues.apache.org/jira/browse/GIRAPH-43
> Project: Giraph
> Issue Type: Bug
> Components: graph
> Affects Versions: 0.70.0
> Reporter: Severin Corsten
> Priority: Minor
>
> I have got a problem regarding Counters:
> For testing and benchmarking I added a lot of timers bud now the task fails with this Exception:
> org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120
> I don't understand why this happens. Within GiraphJob the counter limit is set to 512.
> In the cluster configuration there is no value specified.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-43) GiraphJob: Counter Limit is not set
Posted by "Jakob Homan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-43?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118635#comment-13118635 ]
Jakob Homan commented on GIRAPH-43:
-----------------------------------
Thinking about this some more, 120 is a pretty low limit, particularly since this new limitation was created after people were creating more than a million counters per job. I'd probably suggest adding a note to the documentation that when using 203 or up to make sure the limit is up'ed to a reasonable value. I'll open a new JIRA for it.
> GiraphJob: Counter Limit is not set
> -----------------------------------
>
> Key: GIRAPH-43
> URL: https://issues.apache.org/jira/browse/GIRAPH-43
> Project: Giraph
> Issue Type: Bug
> Components: graph
> Affects Versions: 0.70.0
> Reporter: Severin Corsten
> Priority: Minor
>
> I have got a problem regarding Counters:
> For testing and benchmarking I added a lot of timers bud now the task fails with this Exception:
> org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120
> I don't understand why this happens. Within GiraphJob the counter limit is set to 512.
> In the cluster configuration there is no value specified.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (GIRAPH-43) GiraphJob: Counter Limit
is not set
Posted by "Avery Ching (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-43?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117891#comment-13117891 ]
Avery Ching edited comment on GIRAPH-43 at 9/30/11 5:29 AM:
------------------------------------------------------------
Yeah, this kind of sucks. One thing we could do is say keep the timers (counters) only for the last n supersteps. This way we don't grow infinitely. I.e use two counters per superstep
superstep a = 3
superstep a milliseconds = 70
superstep b = 4
superstep b milliseconds = 71
...
For jobs with a lot of supersteps, we can't just grow forever given this limitation.
was (Author: aching):
Yeah, this kind of sucks. One thing we could do is say keep the timers (counters) only for the last n supersteps. This way we don't grow infinitely. I.e use two counters per superstep
superstep a = 3
superstep a milliseconds = 70
superstep b = 4
superstep b milliseconds = 71
...
For jobs with a lot of supersteps, we can just grow forever given this limitation.
> GiraphJob: Counter Limit is not set
> -----------------------------------
>
> Key: GIRAPH-43
> URL: https://issues.apache.org/jira/browse/GIRAPH-43
> Project: Giraph
> Issue Type: Bug
> Components: graph
> Affects Versions: 0.70.0
> Reporter: Severin Corsten
> Priority: Minor
>
> I have got a problem regarding Counters:
> For testing and benchmarking I added a lot of timers bud now the task fails with this Exception:
> org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120
> I don't understand why this happens. Within GiraphJob the counter limit is set to 512.
> In the cluster configuration there is no value specified.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-43) GiraphJob: Counter Limit is not set
Posted by "Dmitriy V. Ryaboy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-43?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117855#comment-13117855 ]
Dmitriy V. Ryaboy commented on GIRAPH-43:
-----------------------------------------
what version of hadoop are you running? Counter limits are fairly new. I wonder if there's a default of 120.
> GiraphJob: Counter Limit is not set
> -----------------------------------
>
> Key: GIRAPH-43
> URL: https://issues.apache.org/jira/browse/GIRAPH-43
> Project: Giraph
> Issue Type: Bug
> Components: graph
> Affects Versions: 0.70.0
> Reporter: Severin Corsten
> Priority: Minor
>
> I have got a problem regarding Counters:
> For testing and benchmarking I added a lot of timers bud now the task fails with this Exception:
> org.apache.hadoop.mapred.Counters$CountersExceededException: Error: Exceeded limits on number of counters - Counters=120 Limit=120
> I don't understand why this happens. Within GiraphJob the counter limit is set to 512.
> In the cluster configuration there is no value specified.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira