You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Zhiwei Gu (Created) (JIRA)" <ji...@apache.org> on 2011/10/14 02:16:12 UTC
[jira] [Created] (GIRAPH-52) There should be a scheme to limit the
counter
There should be a scheme to limit the counter
---------------------------------------------
Key: GIRAPH-52
URL: https://issues.apache.org/jira/browse/GIRAPH-52
Project: Giraph
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.70.0
Reporter: Zhiwei Gu
Fix For: 0.70.0
For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-52) There should be a scheme to limit
the counter
Posted by "Zhiwei Gu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127291#comment-13127291 ]
Zhiwei Gu commented on GIRAPH-52:
---------------------------------
what about add a CounterManager to do centralized management of the counters?
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.70.0
> Reporter: Zhiwei Gu
> Fix For: 0.70.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-52) There should be a scheme to limit
the counter
Posted by "Avery Ching (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127301#comment-13127301 ]
Avery Ching commented on GIRAPH-52:
-----------------------------------
Anything is probably better than the adhoc counter system we have now. =) I would certainly agree that a CounterManager concept that is cleaner would be nice to have and keep us within the limits of how many counters are available. I personally would vote for the last n iterations (sliding window) approach.
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.70.0
> Reporter: Zhiwei Gu
> Fix For: 0.70.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-52) There should be a scheme to limit
the counter
Posted by "Zhiwei Gu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127746#comment-13127746 ]
Zhiwei Gu commented on GIRAPH-52:
---------------------------------
Thank you Avery, I'll implement the CounterManager using the sliding window approach.
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.70.0
> Reporter: Zhiwei Gu
> Fix For: 0.70.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-52) There should be a scheme to limit
the counter
Posted by "Jakob Homan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127235#comment-13127235 ]
Jakob Homan commented on GIRAPH-52:
-----------------------------------
If you'd like to come up with a way around the limit, this would be the correct JIRA. Perhaps just output statistics for the last n iterations? Or every m iteration, skipping enough to stay under the limit? One could check for the presence of the conf value with reasonable certainty that if it's not there this behavior would be appropriate (this isn't full proof since the value could be set on the server but not the client, but this would work 99% of the time and an error message, if it doesn't, could suggest that possibility).
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.70.0
> Reporter: Zhiwei Gu
> Fix For: 0.70.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-52) There should be a scheme to limit
the counter
Posted by "Jakob Homan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127239#comment-13127239 ]
Jakob Homan commented on GIRAPH-52:
-----------------------------------
above, last as in final, not previous.
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.70.0
> Reporter: Zhiwei Gu
> Fix For: 0.70.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-52) There should be a scheme to limit
the counter
Posted by "Avery Ching (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127302#comment-13127302 ]
Avery Ching commented on GIRAPH-52:
-----------------------------------
By the way Zhiwei, I've added you as a contributor so you can assign this to yourself if you like. =)
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.70.0
> Reporter: Zhiwei Gu
> Fix For: 0.70.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-52) There should be a scheme to limit
the counter
Posted by "Jakob Homan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127299#comment-13127299 ]
Jakob Homan commented on GIRAPH-52:
-----------------------------------
That would be quite reasonable. It could implement either of the schemes above (or a different one), or for non-limited clusters just churn out the counters.
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.70.0
> Reporter: Zhiwei Gu
> Fix For: 0.70.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-52) There should be a scheme to limit
the counter
Posted by "Avery Ching (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127308#comment-13127308 ]
Avery Ching commented on GIRAPH-52:
-----------------------------------
To implement sliding windows we can be a little tricky.
counter_1 = superstep of counter 1
counter_1_val = time of superstep of counter 1
counter_2 = superstep of counter 2
counter_2_val = time of superstep of counter 2
In this way, we have to use 2n counters to display the last n supersteps, but we need not remove any counters.
Agreed on the MR2 work.
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.70.0
> Reporter: Zhiwei Gu
> Fix For: 0.70.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-52) There should be a scheme to limit the
counter
Posted by "Jakob Homan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated GIRAPH-52:
------------------------------
Fix Version/s: (was: 0.1.0)
0.2.0
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.1.0
> Reporter: Zhiwei Gu
> Assignee: Zhiwei Gu
> Fix For: 0.2.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-52) There should be a scheme to limit
the counter
Posted by "Avery Ching (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127169#comment-13127169 ]
Avery Ching commented on GIRAPH-52:
-----------------------------------
Here's a related issue https://issues.apache.org/jira/browse/GIRAPH-43.
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.70.0
> Reporter: Zhiwei Gu
> Fix For: 0.70.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-52) There should be a scheme to limit
the counter
Posted by "Jakob Homan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127304#comment-13127304 ]
Jakob Homan commented on GIRAPH-52:
-----------------------------------
I'm not sure sliding window will work, since there's no way to remote counters once they're set. We're limited to n total for the whole job.
Also, for the MR2 work, it would be good to code against an interface so that once MR and its notions of counters are gone, we can have a different implementation in the new framework (although at that point, we can remove this limit itself).
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.70.0
> Reporter: Zhiwei Gu
> Fix For: 0.70.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-52) There should be a scheme to limit
the counter
Posted by "Zhiwei Gu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127234#comment-13127234 ]
Zhiwei Gu commented on GIRAPH-52:
---------------------------------
yeah, but it is resolved as not to fix. Can I reopen it as it is a blocker for me?
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.70.0
> Reporter: Zhiwei Gu
> Fix For: 0.70.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-52) There should be a scheme to limit the
counter
Posted by "Avery Ching (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching reassigned GIRAPH-52:
---------------------------------
Assignee: Zhiwei Gu
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.70.0
> Reporter: Zhiwei Gu
> Assignee: Zhiwei Gu
> Fix For: 0.70.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-52) There should be a scheme to limit
the counter
Posted by "Maja Kabiljo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404780#comment-13404780 ]
Maja Kabiljo commented on GIRAPH-52:
------------------------------------
I see this task was last commented long time ago. Zhiwei, have you started working on it? If not, I'd like to do it.
> There should be a scheme to limit the counter
> ---------------------------------------------
>
> Key: GIRAPH-52
> URL: https://issues.apache.org/jira/browse/GIRAPH-52
> Project: Giraph
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.1.0
> Reporter: Zhiwei Gu
> Assignee: Zhiwei Gu
> Fix For: 0.2.0
>
>
> For hadoop version above 0.20.203.0., the cluster-wise configuration mapreduce.job.counters.limit cannot be overrided, while the superstep iterations is not deterministic, the job might run several hundreds or even thousand of supersteps, it will always kill the job. This will limit the usage of Giraph and is tooooo bad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira