You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Alessandro Presta (JIRA)" <ji...@apache.org> on 2012/06/29 17:39:42 UTC
[jira] [Created] (GIRAPH-224) Server-side combiner
Alessandro Presta created GIRAPH-224:
----------------------------------------
Summary: Server-side combiner
Key: GIRAPH-224
URL: https://issues.apache.org/jira/browse/GIRAPH-224
Project: Giraph
Issue Type: Improvement
Components: bsp
Reporter: Alessandro Presta
Assignee: Alessandro Presta
Currently, if a VertexCombiner is defined in the Configuration, we use it on the client in order to optimize bandwidth and memory usage.
On the server side, we can easily benefit in terms of memory by exploiting the combiner once more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-224) Server-side combiner
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alessandro Presta updated GIRAPH-224:
-------------------------------------
Attachment: GIRAPH-224-1.patch
Updated patch handles combiner exceptions as in the client-side SendMessageCache, i.e. failing with IllegalStateException.
> Server-side combiner
> --------------------
>
> Key: GIRAPH-224
> URL: https://issues.apache.org/jira/browse/GIRAPH-224
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-224-1.patch, GIRAPH-224.patch
>
>
> Currently, if a VertexCombiner is defined in the Configuration, we use it on the client in order to optimize bandwidth and memory usage.
> On the server side, we can easily benefit in terms of memory by exploiting the combiner once more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-224) Server-side combiner
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alessandro Presta updated GIRAPH-224:
-------------------------------------
Attachment: GIRAPH-224-4.patch
> Server-side combiner
> --------------------
>
> Key: GIRAPH-224
> URL: https://issues.apache.org/jira/browse/GIRAPH-224
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-224-1.patch, GIRAPH-224-2.patch, GIRAPH-224-3.patch, GIRAPH-224-4.patch, GIRAPH-224.patch
>
>
> Currently, if a VertexCombiner is defined in the Configuration, we use it on the client in order to optimize bandwidth and memory usage.
> On the server side, we can easily benefit in terms of memory by exploiting the combiner once more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-224) Server-side combiner
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405447#comment-13405447 ]
Avery Ching commented on GIRAPH-224:
------------------------------------
Alessandro, rather than do combining after the messages are all accumulated (already used all our heap to store them), can you do it earlier? Say in SendPartitionMessagesRequest? This will prevent the memory build up.
> Server-side combiner
> --------------------
>
> Key: GIRAPH-224
> URL: https://issues.apache.org/jira/browse/GIRAPH-224
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-224-1.patch, GIRAPH-224.patch
>
>
> Currently, if a VertexCombiner is defined in the Configuration, we use it on the client in order to optimize bandwidth and memory usage.
> On the server side, we can easily benefit in terms of memory by exploiting the combiner once more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-224) Server-side combiner
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alessandro Presta updated GIRAPH-224:
-------------------------------------
Attachment: GIRAPH-224-2.patch
Combining messages in SendPartitionMessagesRequest#doRequest().
I'm doing this in the synchronized block, combining all existing messages with the ones in the request.
Here's how I tested it:
- added a MinimumDoubleCombiner to the simple shortest paths example
- disabled combining on the client side (SendMessageCache)
- logged the number of messages received by every vertex
- verified that it's always 0 or 1
> Server-side combiner
> --------------------
>
> Key: GIRAPH-224
> URL: https://issues.apache.org/jira/browse/GIRAPH-224
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-224-1.patch, GIRAPH-224-2.patch, GIRAPH-224.patch
>
>
> Currently, if a VertexCombiner is defined in the Configuration, we use it on the client in order to optimize bandwidth and memory usage.
> On the server side, we can easily benefit in terms of memory by exploiting the combiner once more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-224) Server-side combiner
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alessandro Presta updated GIRAPH-224:
-------------------------------------
Attachment: GIRAPH-224-3.patch
Previous patch had a bug: I wasn't putting the combined messages back in transientMessages.
The reason why the test seemed to work is because our unit tests use Hadoop RPC by default, and I had no idea this was Netty-specific code.
> Server-side combiner
> --------------------
>
> Key: GIRAPH-224
> URL: https://issues.apache.org/jira/browse/GIRAPH-224
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-224-1.patch, GIRAPH-224-2.patch, GIRAPH-224-3.patch, GIRAPH-224.patch
>
>
> Currently, if a VertexCombiner is defined in the Configuration, we use it on the client in order to optimize bandwidth and memory usage.
> On the server side, we can easily benefit in terms of memory by exploiting the combiner once more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-224) Server-side combiner
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alessandro Presta updated GIRAPH-224:
-------------------------------------
Attachment: GIRAPH-224.patch
> Server-side combiner
> --------------------
>
> Key: GIRAPH-224
> URL: https://issues.apache.org/jira/browse/GIRAPH-224
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-224.patch
>
>
> Currently, if a VertexCombiner is defined in the Configuration, we use it on the client in order to optimize bandwidth and memory usage.
> On the server side, we can easily benefit in terms of memory by exploiting the combiner once more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-224) Server-side combiner
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405448#comment-13405448 ]
Avery Ching commented on GIRAPH-224:
------------------------------------
To be more clear, in the doRequest() method.
> Server-side combiner
> --------------------
>
> Key: GIRAPH-224
> URL: https://issues.apache.org/jira/browse/GIRAPH-224
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-224-1.patch, GIRAPH-224.patch
>
>
> Currently, if a VertexCombiner is defined in the Configuration, we use it on the client in order to optimize bandwidth and memory usage.
> On the server side, we can easily benefit in terms of memory by exploiting the combiner once more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-224) Netty server-side combiner
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching updated GIRAPH-224:
-------------------------------
Summary: Netty server-side combiner (was: Server-side combiner)
> Netty server-side combiner
> --------------------------
>
> Key: GIRAPH-224
> URL: https://issues.apache.org/jira/browse/GIRAPH-224
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-224-1.patch, GIRAPH-224-2.patch, GIRAPH-224-3.patch, GIRAPH-224-4.patch, GIRAPH-224.patch
>
>
> Currently, if a VertexCombiner is defined in the Configuration, we use it on the client in order to optimize bandwidth and memory usage.
> On the server side, we can easily benefit in terms of memory by exploiting the combiner once more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-224) Netty server-side combiner
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415756#comment-13415756 ]
Hudson commented on GIRAPH-224:
-------------------------------
Integrated in Giraph-trunk-Commit #148 (See [https://builds.apache.org/job/Giraph-trunk-Commit/148/])
GIRAPH-224: Netty server-side combiner (apresta via aching). (Revision 1362298)
Result = SUCCESS
aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362298
Files :
* /giraph/trunk/CHANGELOG
* /giraph/trunk/src/main/java/org/apache/giraph/comm/NettyWorkerServer.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/SendPartitionMessagesRequest.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/ServerData.java
* /giraph/trunk/src/test/java/org/apache/giraph/comm/ConnectionTest.java
* /giraph/trunk/src/test/java/org/apache/giraph/comm/RequestTest.java
> Netty server-side combiner
> --------------------------
>
> Key: GIRAPH-224
> URL: https://issues.apache.org/jira/browse/GIRAPH-224
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-224-1.patch, GIRAPH-224-2.patch, GIRAPH-224-3.patch, GIRAPH-224-4.patch, GIRAPH-224.patch
>
>
> Currently, if a VertexCombiner is defined in the Configuration, we use it on the client in order to optimize bandwidth and memory usage.
> On the server side, we can easily benefit in terms of memory by exploiting the combiner once more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-224) Server-side combiner
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403978#comment-13403978 ]
Alessandro Presta commented on GIRAPH-224:
------------------------------------------
I did this in BspServiceWorker, let me know if it's right or I misunderstood the task.
Had to handle IOException from combine(), even though no implementation currently throws that.
I guess it's there for potential implementations where messages are read from disk?
> Server-side combiner
> --------------------
>
> Key: GIRAPH-224
> URL: https://issues.apache.org/jira/browse/GIRAPH-224
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-224.patch
>
>
> Currently, if a VertexCombiner is defined in the Configuration, we use it on the client in order to optimize bandwidth and memory usage.
> On the server side, we can easily benefit in terms of memory by exploiting the combiner once more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-224) Server-side combiner
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415131#comment-13415131 ]
Alessandro Presta commented on GIRAPH-224:
------------------------------------------
Review: https://reviews.apache.org/r/5986/
> Server-side combiner
> --------------------
>
> Key: GIRAPH-224
> URL: https://issues.apache.org/jira/browse/GIRAPH-224
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-224-1.patch, GIRAPH-224-2.patch, GIRAPH-224-3.patch, GIRAPH-224.patch
>
>
> Currently, if a VertexCombiner is defined in the Configuration, we use it on the client in order to optimize bandwidth and memory usage.
> On the server side, we can easily benefit in terms of memory by exploiting the combiner once more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira