You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Bo Wang (JIRA)" <ji...@apache.org> on 2012/05/05 20:59:50 UTC
[jira] [Updated] (GIRAPH-185) Improve concurrency of putMsg /
putMsgList
[ https://issues.apache.org/jira/browse/GIRAPH-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bo Wang updated GIRAPH-185:
---------------------------
Attachment: GIRAPH-185.patch
Sorry for replying late. I've been pretty busy last week with midterms and deadlines. I did a few tests on a 24-core 32GB DRAM machine.
Quite astonishing, the performance of the new version (ConcurrentHashMap + ConcurrentLinkedQueue) is kind of slower than the original version. I ran the PageRankBenchmark with "-e 100 -s 10 -V 10000". Results are as following.
(original)
#cores superstep.exec.time
12 11580
8 10651
4 12944
2 19334
1 20782
(ConcurrentHashMap + ConcurrentLinkedQueue)
#cores superstep.exec.time
12 11639
8 11527
4 13653
2 20354
1 27825
I think the overhead came from the lock/unlock on the get(), especially when reading messages sent in the previous stage where no lock is needed since it's sequential.
To verify my assumption, I changed the ConcurrentLinkedQueue back to ArrayList and wrap it with synchronized, then the performance does improve.
(ConcurrentHashMap)
#cores superstep.exec.time
12 11380
8 11618
4 12834
2 18417
1 22952
In comparison, it's more or less the same as the original version. Then I ran with another set of parameters "-e 1000 -s 10 -V 1000". It seems the fine grain lock does help.
(original)
#cores superstep.exec.time
12 97101
6 11321
(ConcurrentHashMap)
#cores superstep.exec.time
12 92848 (4.4%)
6 10834 (4.3%)
I attached the new patch (ConcurrentHashMap version) for the review.
Claudio, thanks for offering to help. You may checkout a clean version and apply the patch ($ patch -p0 -i GIRAPH-185.patch) to test it on the cluster.
> Improve concurrency of putMsg / putMsgList
> ------------------------------------------
>
> Key: GIRAPH-185
> URL: https://issues.apache.org/jira/browse/GIRAPH-185
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Affects Versions: 0.2.0
> Reporter: Bo Wang
> Assignee: Bo Wang
> Fix For: 0.2.0
>
> Attachments: GIRAPH-185.patch, GIRAPH-185.patch, GIRAPH-185.patch, GIRAPH-185.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Currently in putMsg / putMsgList, a synchronized closure is used to protect the whole transientInMessages when adding the new message. This lock prevents other concurrent calls to putMsg/putMsgList and increases the response time. We should use fine-grain locks to allow high concurrency in message communication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira