You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Maja Kabiljo <ma...@fb.com> on 2012/10/29 18:19:57 UTC
Review Request: Improve the way we keep outgoing messages

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7764/
-----------------------------------------------------------

Review request for giraph.


Description
-------

As per discussion on GIRAPH-357, in standard application chances that we get to use client-side combiner are very low. I experimented with benefits which we can get from not having the client-side combiner at all. It turns out that having a lot of maps in SendMessageCache, and then collection inside each of them, really hurts the performance. 


This addresses bug GIRAPH-388.
    https://issues.apache.org/jira/browse/GIRAPH-388


Diffs
-----

  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/SendMessageCache.java 1402328 
  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/VertexIdMessageCollection.java PRE-CREATION 
  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStoreByPartition.java 1402328 
  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/MessageStoreByPartition.java 1402328 
  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1402328 
  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 1402328 
  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java 1402328 
  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1402328 
  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1402328 

Diff: https://reviews.apache.org/r/7764/diff/


Testing
-------

Single node and pseudo-distributed tests.
Benchmark results: https://issues.apache.org/jira/browse/GIRAPH-388?focusedCommentId=13484591


Thanks,

Maja Kabiljo