You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Avery Ching <av...@gmail.com> on 2011/12/14 20:48:52 UTC
Review Request: GIRAPH-57 Add new RPC call (putVertexIdMessagesList) to
batch putMsgList RPCs together
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3201/
-----------------------------------------------------------
Review request for giraph.
Summary
-------
Added new putVertexIdMessagesList RPC and supporting classes (VertexIdMessages and VertexIdMessagesList) to reduce the total number of RPCs during a flush. This improves the number of RPC / sec and overall I/O bandwidth. The amount of batching is done by the total number of messages and is configurable at runtime (default of 5000, weighted toward helping smaller messages). I have noted some performance results in https://issues.apache.org/jira/browse/GIRAPH-57 (between 25 - 1075 percent improvements).
Also, while tinkering with BasicRPCCommunications, notices inconsistent spaces between 'synchronized' and '('. Removed spaces and standardized in the CODE_CONVENTIONS.
This addresses bug GIRAPH-57.
https://issues.apache.org/jira/browse/GIRAPH-57
Diffs
-----
http://svn.apache.org/repos/asf/incubator/giraph/trunk/CODE_CONVENTIONS 1213849
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ArrayListWritable.java 1213849
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1214406
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java 1213849
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java 1213849
Diff: https://reviews.apache.org/r/3201/diff
Testing
-------
Passed local and Hadoop unittests. Used the RandomMessageBenchmark on a small cluster.
Thanks,
Avery
Re: Review Request: GIRAPH-57 Add new RPC call (putVertexIdMessagesList) to
batch putMsgList RPCs together
Posted by Claudio Martella <cl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3201/#review3928
-----------------------------------------------------------
Ship it!
Looks cool and straight from here. All those CODE_CONVENTIONS fixes made it looks scary from far.
- Claudio
On 2011-12-14 19:50:29, Avery Ching wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3201/
> -----------------------------------------------------------
>
> (Updated 2011-12-14 19:50:29)
>
>
> Review request for giraph.
>
>
> Summary
> -------
>
> Added new putVertexIdMessagesList RPC and supporting classes (VertexIdMessages and VertexIdMessagesList) to reduce the total number of RPCs during a flush. This improves the number of RPC / sec and overall I/O bandwidth. The amount of batching is done by the total number of messages and is configurable at runtime (default of 5000, weighted toward helping smaller messages). I have noted some performance results in https://issues.apache.org/jira/browse/GIRAPH-57 (between 25 - 1075 percent improvements).
>
> Also, while tinkering with BasicRPCCommunications, notices inconsistent spaces between 'synchronized' and '('. Removed spaces and standardized in the CODE_CONVENTIONS.
>
>
> This addresses bug GIRAPH-57.
> https://issues.apache.org/jira/browse/GIRAPH-57
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/CODE_CONVENTIONS 1213849
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ArrayListWritable.java 1213849
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1214406
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java 1213849
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java PRE-CREATION
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java PRE-CREATION
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java 1213849
>
> Diff: https://reviews.apache.org/r/3201/diff
>
>
> Testing
> -------
>
> Passed local and Hadoop unittests. Used the RandomMessageBenchmark on a small cluster.
>
>
> Thanks,
>
> Avery
>
>
Re: Review Request: GIRAPH-57 Add new RPC call (putVertexIdMessagesList) to
batch putMsgList RPCs together
Posted by Avery Ching <av...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3201/
-----------------------------------------------------------
(Updated 2011-12-14 19:50:29.358317)
Review request for giraph.
Changes
-------
Forgot a few important files (VertexIdMessages.java and VertexIdMessagesList.java)
Summary
-------
Added new putVertexIdMessagesList RPC and supporting classes (VertexIdMessages and VertexIdMessagesList) to reduce the total number of RPCs during a flush. This improves the number of RPC / sec and overall I/O bandwidth. The amount of batching is done by the total number of messages and is configurable at runtime (default of 5000, weighted toward helping smaller messages). I have noted some performance results in https://issues.apache.org/jira/browse/GIRAPH-57 (between 25 - 1075 percent improvements).
Also, while tinkering with BasicRPCCommunications, notices inconsistent spaces between 'synchronized' and '('. Removed spaces and standardized in the CODE_CONVENTIONS.
This addresses bug GIRAPH-57.
https://issues.apache.org/jira/browse/GIRAPH-57
Diffs (updated)
-----
http://svn.apache.org/repos/asf/incubator/giraph/trunk/CODE_CONVENTIONS 1213849
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ArrayListWritable.java 1213849
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1214406
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java 1213849
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java PRE-CREATION
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java PRE-CREATION
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java 1213849
Diff: https://reviews.apache.org/r/3201/diff
Testing
-------
Passed local and Hadoop unittests. Used the RandomMessageBenchmark on a small cluster.
Thanks,
Avery