You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Hyunsik Choi (Created) (JIRA)" <ji...@apache.org> on 2011/09/28 07:35:46 UTC
[jira] [Created] (GIRAPH-42) The MapReduce counter 'Sent Messages'
doesn't work.
The MapReduce counter 'Sent Messages' doesn't work.
---------------------------------------------------
Key: GIRAPH-42
URL: https://issues.apache.org/jira/browse/GIRAPH-42
Project: Giraph
Issue Type: Bug
Components: bsp
Reporter: Hyunsik Choi
Priority: Minor
The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
{noformat}
.
.
11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished vertices=600000
11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=600000
.
.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-42) The MapReduce counter 'Sent Messages'
doesn't work.
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen updated GIRAPH-42:
------------------------------
Attachment: GIRAPH-42.patch
trivial patch for messages counters.
> The MapReduce counter 'Sent Messages' doesn't work.
> ---------------------------------------------------
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
> Reporter: Hyunsik Choi
> Priority: Minor
> Attachments: GIRAPH-42.patch
>
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished vertices=600000
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=600000
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-42) The MapReduce counter 'Sent
Messages' doesn't work.
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479206#comment-13479206 ]
Doron Cohen commented on GIRAPH-42:
-----------------------------------
Hi, now I am not sure that my first patch here addresses the problem discussed in this issue.
The problem I saw is that the sent messages counter is always 0, and
I think the cause for always-zero-count are these lines in BspServiceMaster:
{code}
updateCounters(GlobalStats globalStats) {
...
sentMessagesCounter.increment(
globalStats.getMessageCount() -
sentMessagesCounter.getValue());
}
{code}
The thing is that globalStats messages-count is updated via calls to WorkerClientRequestProcessor.resetMessageCount().
Since resetMessageCount() sets the local counter to 0, it seems the current code simply shows the number of messages
sent in the last superstep, Which is always 0, I think.
So, it is wrong to subtract sentMessagesCounter.getValue(), instead, just need to add globalStats.getMessageCount().
I think this issue is originally about something else - how to consider combined messages in the messages counts.
>From what I could understand in the code the current approach is (1):
bq. 1. Increment the Counter in the sentMsg Method in Vertex -> If messages are combined it does not show up in the Counter.
(anyhow I could not find where the code attempts to do (3).)
Updated patch to follow, assuming this problem should be fixed i this issue, or is a new issue required?
> The MapReduce counter 'Sent Messages' doesn't work.
> ---------------------------------------------------
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
> Reporter: Hyunsik Choi
> Priority: Minor
> Attachments: GIRAPH-42.patch
>
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished vertices=600000
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=600000
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-42) The MapReduce counter 'Sent
Messages' doesn't work.
Posted by "Severin Corsten (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116775#comment-13116775 ]
Severin Corsten commented on GIRAPH-42:
---------------------------------------
I don't see a reson why not to implement both 1 and 3. That allows the user to have a better view on his Combiner.
> The MapReduce counter 'Sent Messages' doesn't work.
> ---------------------------------------------------
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
> Reporter: Hyunsik Choi
> Priority: Minor
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished vertices=600000
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=600000
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-42) The MapReduce counter 'Sent
Messages' doesn't work.
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474444#comment-13474444 ]
Doron Cohen commented on GIRAPH-42:
-----------------------------------
I tried the PageRankBenchmark example it it shows 0 for 'Sent messages' counter, so this is still a problem?
Looking at BspServiceMaster, I think that since the messages counters are being reset() by BspServiceWorker, the master should just add the obtained counter? I applied this fix and the messages seems correct now - not 0 for sure. I'll add a trivial patch.
> The MapReduce counter 'Sent Messages' doesn't work.
> ---------------------------------------------------
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
> Reporter: Hyunsik Choi
> Priority: Minor
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished vertices=600000
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=600000
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-42) The MapReduce counter 'Sent
Messages' doesn't work.
Posted by "Severin Corsten (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116928#comment-13116928 ]
Severin Corsten commented on GIRAPH-42:
---------------------------------------
In addition I found an issue implementing point 1 and 3 together, thus I am not quite sure whether its issued by it:
I run the SimpleShortestPathVertex in a standard Giraph on a big dataset and got as a result: Giraph Stats: Superstep: 15
After adding a counter in Vertex.sendMsg and one in the GiraphMapper just behind the compute call, the result changed to: Giraph Stats: Superstep: 16 while the number of supersteps show in Giraph Timers is the same (15).
I am not sure whether is caused by an other change or by a bad condition and I didn't reproduce it yet.
> The MapReduce counter 'Sent Messages' doesn't work.
> ---------------------------------------------------
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
> Reporter: Hyunsik Choi
> Priority: Minor
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished vertices=600000
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=600000
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-42) The MapReduce counter 'Sent
Messages' doesn't work.
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485982#comment-13485982 ]
Doron Cohen commented on GIRAPH-42:
-----------------------------------
Hi [~aching], thanks for reviewing this.
{quote}
The sent messages counter is per superstep, so perhaps what would be better is to add a total sent messages counter to sum up all the messages sent during all the supersteps.
{quote}
Oh, I was not aware of this, I'll work a new patch for this.
Doron
> The MapReduce counter 'Sent Messages' doesn't work.
> ---------------------------------------------------
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
> Affects Versions: 0.1.0
> Reporter: Hyunsik Choi
> Priority: Minor
> Labels: newbie, patch
> Attachments: GIRAPH-42-2.patch, GIRAPH-42.patch
>
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished vertices=600000
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=600000
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-42) The MapReduce counter 'Sent Messages'
doesn't work.
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen updated GIRAPH-42:
------------------------------
Attachment: GIRAPH-42-2.patch
version 2 of the patch.
> The MapReduce counter 'Sent Messages' doesn't work.
> ---------------------------------------------------
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
> Reporter: Hyunsik Choi
> Priority: Minor
> Attachments: GIRAPH-42-2.patch, GIRAPH-42.patch
>
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished vertices=600000
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=600000
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-42) The MapReduce counter 'Sent
Messages' doesn't work.
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485866#comment-13485866 ]
Avery Ching commented on GIRAPH-42:
-----------------------------------
Hi [~doronc], thanks for taking a look at this. The sent messages counter is per superstep, so perhaps what would be better is to add a total sent messages counter to sum up all the messages sent during all the supersteps.
> The MapReduce counter 'Sent Messages' doesn't work.
> ---------------------------------------------------
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
> Affects Versions: 0.1.0
> Reporter: Hyunsik Choi
> Priority: Minor
> Labels: newbie, patch
> Attachments: GIRAPH-42-2.patch, GIRAPH-42.patch
>
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished vertices=600000
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=600000
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-42) The MapReduce counter 'Sent
Messages' doesn't work.
Posted by "Severin Corsten (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116196#comment-13116196 ]
Severin Corsten commented on GIRAPH-42:
---------------------------------------
I looked on this problem today because of an other reason, bud I don't know how to fix it right:
1. Increment the Counter in the sentMsg Method in Vertex -> If messages are combined it does not show up in the Counter.
2. Increment the Counter in after applying the combiner, which is currently impossible because the RPC Connection does not know the contex.
3. Increment the Counter in the map method when executing the next superstep -> Delays the counting by one superstep.
At the moment I like 3 the most.
Greetings Severin
> The MapReduce counter 'Sent Messages' doesn't work.
> ---------------------------------------------------
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
> Reporter: Hyunsik Choi
> Priority: Minor
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished vertices=600000
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=600000
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-42) The MapReduce counter 'Sent
Messages' doesn't work.
Posted by "Hyunsik Choi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116592#comment-13116592 ]
Hyunsik Choi commented on GIRAPH-42:
------------------------------------
I also like the third approach. The current implementation seems to aim at the third approach.
> The MapReduce counter 'Sent Messages' doesn't work.
> ---------------------------------------------------
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
> Reporter: Hyunsik Choi
> Priority: Minor
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished vertices=600000
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=600000
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira