You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Thomas Jungblut (Created) (JIRA)" <ji...@apache.org> on 2011/11/24 18:11:40 UTC
[jira] [Created] (HAMA-479) Add Counters to Hama Jobs
Add Counters to Hama Jobs
-------------------------
Key: HAMA-479
URL: https://issues.apache.org/jira/browse/HAMA-479
Project: Hama
Issue Type: New Feature
Reporter: Thomas Jungblut
Fix For: 0.4.0
We should add counters for each Hama job.
We want to add message counters (bytes transfered, local messages sent, outward messages sent, time in sync etc.), for the last superstep and summed up for all supersteps.
It should be as simple as the from Hadoop provided mechanism with enums.
We could eventually use zookeeper instead of the jobreport.
For the SSSP Example it should be useful that the last superstep counters are always in sync in each task after a superstep. == After barrier left, query zookeeper for the values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HAMA-479) Add Counters to Hama Jobs
Posted by "Edward J. Yoon (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon reassigned HAMA-479:
-----------------------------------
Assignee: Edward J. Yoon
> Add Counters to Hama Jobs
> -------------------------
>
> Key: HAMA-479
> URL: https://issues.apache.org/jira/browse/HAMA-479
> Project: Hama
> Issue Type: New Feature
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Labels: bsp
> Fix For: 0.4.0
>
>
> We should add counters for each Hama job.
> We want to add message counters (bytes transfered, local messages sent, outward messages sent, time in sync etc.), for the last superstep and summed up for all supersteps.
> It should be as simple as the from Hadoop provided mechanism with enums.
> We could eventually use zookeeper instead of the jobreport.
> For the SSSP Example it should be useful that the last superstep counters are always in sync in each task after a superstep. == After barrier left, query zookeeper for the values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-479) Add Counters to Hama Jobs
Posted by "ChiaHung Lin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158408#comment-13158408 ]
ChiaHung Lin commented on HAMA-479:
-----------------------------------
{quote}
+1 for the patch, that is a quite acceptable solution.
The global stuff needs either to be stored in ZK or we are going to propagate this back to the BSPMaster via statusreports.
{quote}
+1
> Add Counters to Hama Jobs
> -------------------------
>
> Key: HAMA-479
> URL: https://issues.apache.org/jira/browse/HAMA-479
> Project: Hama
> Issue Type: New Feature
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Labels: bsp
> Fix For: 0.4.0
>
> Attachments: counters.patch
>
>
> We should add counters for each Hama job.
> We want to add message counters (bytes transfered, local messages sent, outward messages sent, time in sync etc.), for the last superstep and summed up for all supersteps.
> It should be as simple as the from Hadoop provided mechanism with enums.
> We could eventually use zookeeper instead of the jobreport.
> For the SSSP Example it should be useful that the last superstep counters are always in sync in each task after a superstep. == After barrier left, query zookeeper for the values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-479) Add Counters to Hama Jobs
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158402#comment-13158402 ]
Thomas Jungblut commented on HAMA-479:
--------------------------------------
+1 for the patch, that is a quite acceptable solution. ;)
The global stuff needs either to be stored in ZK or we are going to propagate this back to the BSPMaster via statusreports.
> Add Counters to Hama Jobs
> -------------------------
>
> Key: HAMA-479
> URL: https://issues.apache.org/jira/browse/HAMA-479
> Project: Hama
> Issue Type: New Feature
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Labels: bsp
> Fix For: 0.4.0
>
> Attachments: counters.patch
>
>
> We should add counters for each Hama job.
> We want to add message counters (bytes transfered, local messages sent, outward messages sent, time in sync etc.), for the last superstep and summed up for all supersteps.
> It should be as simple as the from Hadoop provided mechanism with enums.
> We could eventually use zookeeper instead of the jobreport.
> For the SSSP Example it should be useful that the last superstep counters are always in sync in each task after a superstep. == After barrier left, query zookeeper for the values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-479) Add Counters to Hama Jobs
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158295#comment-13158295 ]
Edward J. Yoon commented on HAMA-479:
-------------------------------------
Note to myself:
We have to update this page:
http://wiki.apache.org/hama/BSPModel
> Add Counters to Hama Jobs
> -------------------------
>
> Key: HAMA-479
> URL: https://issues.apache.org/jira/browse/HAMA-479
> Project: Hama
> Issue Type: New Feature
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Labels: bsp
> Fix For: 0.4.0
>
>
> We should add counters for each Hama job.
> We want to add message counters (bytes transfered, local messages sent, outward messages sent, time in sync etc.), for the last superstep and summed up for all supersteps.
> It should be as simple as the from Hadoop provided mechanism with enums.
> We could eventually use zookeeper instead of the jobreport.
> For the SSSP Example it should be useful that the last superstep counters are always in sync in each task after a superstep. == After barrier left, query zookeeper for the values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-479) Add Counters to Hama Jobs
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158291#comment-13158291 ]
Edward J. Yoon commented on HAMA-479:
-------------------------------------
{quote}
enums and strings as keys for counters?
{quote}
I'd like to.
> Add Counters to Hama Jobs
> -------------------------
>
> Key: HAMA-479
> URL: https://issues.apache.org/jira/browse/HAMA-479
> Project: Hama
> Issue Type: New Feature
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Labels: bsp
> Fix For: 0.4.0
>
>
> We should add counters for each Hama job.
> We want to add message counters (bytes transfered, local messages sent, outward messages sent, time in sync etc.), for the last superstep and summed up for all supersteps.
> It should be as simple as the from Hadoop provided mechanism with enums.
> We could eventually use zookeeper instead of the jobreport.
> For the SSSP Example it should be useful that the last superstep counters are always in sync in each task after a superstep. == After barrier left, query zookeeper for the values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-479) Add Counters to Hama Jobs
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158279#comment-13158279 ]
Thomas Jungblut commented on HAMA-479:
--------------------------------------
Should we provide both, enums and strings as keys for counters?
At the end they are going to be both in a map as a string I guess, so we should support both of them.
> Add Counters to Hama Jobs
> -------------------------
>
> Key: HAMA-479
> URL: https://issues.apache.org/jira/browse/HAMA-479
> Project: Hama
> Issue Type: New Feature
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Labels: bsp
> Fix For: 0.4.0
>
>
> We should add counters for each Hama job.
> We want to add message counters (bytes transfered, local messages sent, outward messages sent, time in sync etc.), for the last superstep and summed up for all supersteps.
> It should be as simple as the from Hadoop provided mechanism with enums.
> We could eventually use zookeeper instead of the jobreport.
> For the SSSP Example it should be useful that the last superstep counters are always in sync in each task after a superstep. == After barrier left, query zookeeper for the values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-479) Add Counters to Hama Jobs
Posted by "Edward J. Yoon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-479:
--------------------------------
Attachment: counters.patch
Here's my quick and dirty patch.
BTW, updatesMade can be counted by each task's counter but how can we get the global sum counts in each superstep? :/
> Add Counters to Hama Jobs
> -------------------------
>
> Key: HAMA-479
> URL: https://issues.apache.org/jira/browse/HAMA-479
> Project: Hama
> Issue Type: New Feature
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Labels: bsp
> Fix For: 0.4.0
>
> Attachments: counters.patch
>
>
> We should add counters for each Hama job.
> We want to add message counters (bytes transfered, local messages sent, outward messages sent, time in sync etc.), for the last superstep and summed up for all supersteps.
> It should be as simple as the from Hadoop provided mechanism with enums.
> We could eventually use zookeeper instead of the jobreport.
> For the SSSP Example it should be useful that the last superstep counters are always in sync in each task after a superstep. == After barrier left, query zookeeper for the values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HAMA-479) Add Counters to Hama Jobs
Posted by "Edward J. Yoon (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon resolved HAMA-479.
---------------------------------
Resolution: Fixed
Committed!
> Add Counters to Hama Jobs
> -------------------------
>
> Key: HAMA-479
> URL: https://issues.apache.org/jira/browse/HAMA-479
> Project: Hama
> Issue Type: New Feature
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Labels: bsp
> Fix For: 0.4.0
>
> Attachments: counters.patch
>
>
> We should add counters for each Hama job.
> We want to add message counters (bytes transfered, local messages sent, outward messages sent, time in sync etc.), for the last superstep and summed up for all supersteps.
> It should be as simple as the from Hadoop provided mechanism with enums.
> We could eventually use zookeeper instead of the jobreport.
> For the SSSP Example it should be useful that the last superstep counters are always in sync in each task after a superstep. == After barrier left, query zookeeper for the values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-479) Add Counters to Hama Jobs
Posted by "ChiaHung Lin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158405#comment-13158405 ]
ChiaHung Lin commented on HAMA-479:
-----------------------------------
{quote}
BTW, updatesMade can be counted by each task's counter but how can we get the global sum counts in each superstep? :/
{quote}
Maybe storing information in zookeeper would help?
> Add Counters to Hama Jobs
> -------------------------
>
> Key: HAMA-479
> URL: https://issues.apache.org/jira/browse/HAMA-479
> Project: Hama
> Issue Type: New Feature
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Labels: bsp
> Fix For: 0.4.0
>
> Attachments: counters.patch
>
>
> We should add counters for each Hama job.
> We want to add message counters (bytes transfered, local messages sent, outward messages sent, time in sync etc.), for the last superstep and summed up for all supersteps.
> It should be as simple as the from Hadoop provided mechanism with enums.
> We could eventually use zookeeper instead of the jobreport.
> For the SSSP Example it should be useful that the last superstep counters are always in sync in each task after a superstep. == After barrier left, query zookeeper for the values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-479) Add Counters to Hama Jobs
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158267#comment-13158267 ]
Edward J. Yoon commented on HAMA-479:
-------------------------------------
It'd be nice if the supersteps and basic I/O records are built-in counters of BSP framework. Additionally, we should allows users to employ their own counters.
In SSSP Example case, sync() calls can be reduced to 1/3 using counter, maybe as below
{code}
...
peer.incrCounter(MSG_EXCHANGE, String.valueOf(peer.getSuperstepCount()), updatesMade);
....
peer.sync();
// exit from loop
if (peer.getCounter(MSG_EXCHANGE,
String.valueOf(peer.getSuperstepCount() - 1)).getCounter() == 0) {
break;
}
{code}
> Add Counters to Hama Jobs
> -------------------------
>
> Key: HAMA-479
> URL: https://issues.apache.org/jira/browse/HAMA-479
> Project: Hama
> Issue Type: New Feature
> Reporter: Thomas Jungblut
> Assignee: Edward J. Yoon
> Labels: bsp
> Fix For: 0.4.0
>
>
> We should add counters for each Hama job.
> We want to add message counters (bytes transfered, local messages sent, outward messages sent, time in sync etc.), for the last superstep and summed up for all supersteps.
> It should be as simple as the from Hadoop provided mechanism with enums.
> We could eventually use zookeeper instead of the jobreport.
> For the SSSP Example it should be useful that the last superstep counters are always in sync in each task after a superstep. == After barrier left, query zookeeper for the values.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira