You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Avery Ching (JIRA)" <ji...@apache.org> on 2012/07/17 09:27:34 UTC
[jira] [Created] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Avery Ching created GIRAPH-262:
----------------------------------
Summary: Netty optimizations and bandwidth tracking
Key: GIRAPH-262
URL: https://issues.apache.org/jira/browse/GIRAPH-262
Project: Giraph
Issue Type: Improvement
Reporter: Avery Ching
Assignee: Avery Ching
* Makes netty the default instead of HadoopRPC
* Added optimization to handle requests locally when possible rather than go over the network
* Added TimedLogger to print only within a given time period
* Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
* Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
* Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimization to handle
requests locally whenever possible
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430570#comment-13430570 ]
Hudson commented on GIRAPH-262:
-------------------------------
Integrated in Giraph-trunk-Commit #161 (See [https://builds.apache.org/job/Giraph-trunk-Commit/161/])
GIRAPH-288: Bandwidth tracking - subset of GIRAPH-262. (aching) (Revision 1370438)
Result = SUCCESS
aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1370438
Files :
* /giraph/trunk/CHANGELOG
* /giraph/trunk/src/main/java/org/apache/giraph/comm/ByteCounter.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/NettyClient.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/NettyServer.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/RequestDecoder.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/ResponseClientHandler.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/SendVertexRequest.java
* /giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
* /giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
* /giraph/trunk/src/main/java/org/apache/giraph/utils/TimedLogger.java
* /giraph/trunk/src/test/java/org/apache/giraph/comm/ConnectionTest.java
> Netty optimization to handle requests locally whenever possible
> ---------------------------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.4.patch, GIRAPH-262.5.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-262) Netty optimization to handle requests
locally whenever possible
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching updated GIRAPH-262:
-------------------------------
Summary: Netty optimization to handle requests locally whenever possible (was: Netty optimizations and bandwidth tracking)
> Netty optimization to handle requests locally whenever possible
> ---------------------------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.4.patch, GIRAPH-262.5.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman reassigned GIRAPH-262:
----------------------------------
Assignee: Avery Ching (was: Eli Reisman)
oops
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421091#comment-13421091 ]
Avery Ching commented on GIRAPH-262:
------------------------------------
Thanks Jakob.
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching updated GIRAPH-262:
-------------------------------
Attachment: GIRAPH-262.5.patch
Rebased off of GIRAPH-288 and GIRAPH-289. Ready for commit after a +1.
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.4.patch, GIRAPH-262.5.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching updated GIRAPH-262:
-------------------------------
Attachment: GIRAPH-262.patch
I have attached GIRAPH-262.patch and have a reviewboard open for comments (https://reviews.apache.org/r/6008/)
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimization to handle
requests locally whenever possible
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430631#comment-13430631 ]
Hudson commented on GIRAPH-262:
-------------------------------
Integrated in Giraph-trunk-Commit #162 (See [https://builds.apache.org/job/Giraph-trunk-Commit/162/])
GIRAPH-262: Netty optimization to handle requests locally whenever
possible. (aching) (Revision 1370476)
Result = SUCCESS
aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1370476
Files :
* /giraph/trunk/CHANGELOG
* /giraph/trunk/pom.xml
* /giraph/trunk/src/main/java/org/apache/giraph/benchmark/PseudoRandomVertexInputFormat.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/NettyWorkerClient.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/NettyWorkerClientServer.java
> Netty optimization to handle requests locally whenever possible
> ---------------------------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.4.patch, GIRAPH-262.5.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419499#comment-13419499 ]
Avery Ching commented on GIRAPH-262:
------------------------------------
Thanks for checking it out Eli. We will still have future optimizations for Netty, but this is a first step. Can a committer please review this as well?
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Eugene Koontz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429326#comment-13429326 ]
Eugene Koontz commented on GIRAPH-262:
--------------------------------------
Jakob, good point - linking with GIRAPH-211 (which will have a patch available very soon).
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416400#comment-13416400 ]
Avery Ching commented on GIRAPH-262:
------------------------------------
The idea behind ChannelRotator is that we can use multiple connections between clients and servers and rotate the connections to possibly get better bandwidth (i.e. connection limited). I havn't experimented too much with it, but it does appear to make a difference in performance. The default is one connection between workers.
Handling requests locally improves performance nicely for small numbers of workers as 1/n messages do not need to go over the network.
Additionally, I made the send/receive buffer sizes to be tunable as this seems to also impact performance.
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Eugene Koontz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427591#comment-13427591 ]
Eugene Koontz commented on GIRAPH-262:
--------------------------------------
I'll take a look today or tomorrow. I feel like I've got some necessary background on it due to my working on the Netty code in GIRAPH-211.
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429018#comment-13429018 ]
Avery Ching commented on GIRAPH-262:
------------------------------------
Anyone, please? It's fairly straightforward and a free performance boost (given n workers reduces messages by 1/n)! =)
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419310#comment-13419310 ]
Eli Reisman commented on GIRAPH-262:
------------------------------------
Works well for me under a variety of loads, and dropping the netty server-side thread max from 64 to 32 made it overload less on the cluster I test on, so that was a unexpected bonus too! Great stuff!
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416398#comment-13416398 ]
Eli Reisman commented on GIRAPH-262:
------------------------------------
This looks fantastic! Dumb question: what does the ChannelRotator do, or rather what feature is it providing?
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418781#comment-13418781 ]
Eli Reisman commented on GIRAPH-262:
------------------------------------
nice, I was thinking of the local message change myself for a JIRA!
After playing with GIRAPH-256 a bunch and trying to use that to regulate the size of outgoing collections of vertices after being read from input splits, I am seeing that allowing too big a message to go through or too many small ones both overwhelm the netty buffers. Would it be useful to find a way to weigh outgoing objects of every sort and simply tune the netty buffers to handle those sizes? That, and get them out of serialization before they pile up? Perhaps this (before the calculations but during message passing or vertex passing at any super step) is where spill to disk stuff could be handy? I am having a lot of trouble getting the settings right and consistent to get through different data loads with my memory constraints.
I'll try this patch for sure and see if it helps, the rotator might be just the thing to stop message pile ups. I keep seeing netty get overwhelmed on the Decode side.
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Eugene Koontz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429290#comment-13429290 ]
Eugene Koontz commented on GIRAPH-262:
--------------------------------------
Hi Avery,
It seems like the patch has gone stale:
{code}
ekoontz@mac ~/giraph $ git log -1
commit f8f16bd768557b66ba459b3f5ba7ccc437d2191a
Author: Avery Ching <ac...@apache.org>
Date: Sun Aug 5 00:17:12 2012 +0000
GIRAPH-45: Improve the way to keep outgoing messages (majakabiljo via
aching).
ekoontz@mac ~/giraph $ patch -p0 < GIRAPH-262.2.patch
...
patching file src/main/java/org/apache/giraph/comm/NettyWorkerClient.java
Hunk #1 succeeded at 86 (offset 3 lines).
Hunk #2 succeeded at 108 (offset 3 lines).
Hunk #3 succeeded at 172 with fuzz 2 (offset 3 lines).
Hunk #4 succeeded at 210 (offset 3 lines).
Hunk #5 FAILED at 227.
Hunk #6 succeeded at 280 (offset 33 lines).
Hunk #7 succeeded at 374 (offset 34 lines).
Hunk #8 succeeded at 387 (offset 34 lines).
1 out of 8 hunks FAILED -- saving rejects to file src/main/java/org/apache/giraph/comm/NettyWorkerClient.java.rej
...
patching file src/main/java/org/apache/giraph/graph/BspUtils.java
Hunk #1 FAILED at 420.
1 out of 1 hunk FAILED -- saving rejects to file src/main/java/org/apache/giraph/graph/BspUtils.java.rej
...
{code}
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching updated GIRAPH-262:
-------------------------------
Attachment: GIRAPH-262.4.patch
Updated as per https://reviews.apache.org/r/6008/
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.4.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching updated GIRAPH-262:
-------------------------------
Attachment: GIRAPH-262.3.patch
Sorry Eugene, here's a freshly rebased one.
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421078#comment-13421078 ]
Jakob Homan commented on GIRAPH-262:
------------------------------------
I'll take a look.
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman reassigned GIRAPH-262:
----------------------------------
Assignee: Eli Reisman (was: Avery Ching)
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Eli Reisman
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimization to handle
requests locally whenever possible
Posted by "Eugene Koontz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430577#comment-13430577 ]
Eugene Koontz commented on GIRAPH-262:
--------------------------------------
+1. Passes mvn clean verify with no -P (Hadoop 0.20.203.0) and with -Phadoop_trunk.
> Netty optimization to handle requests locally whenever possible
> ---------------------------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.4.patch, GIRAPH-262.5.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429321#comment-13429321 ]
Jakob Homan commented on GIRAPH-262:
------------------------------------
Sorry I've not been able to do a full review, but I don't think we can make netty the default yet for secure clusters. That opens a big security hole for incorrectly configured grids...
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching updated GIRAPH-262:
-------------------------------
Attachment: GIRAPH-262.2.patch
Rebasing. Can a committer please take a look?
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-262) Netty optimizations and bandwidth
tracking
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429332#comment-13429332 ]
Eli Reisman commented on GIRAPH-262:
------------------------------------
I think this would complement/improve performance for GIRAPH-275 as well, I will patch this in and test against that build to see what happens. Thanks again for doing this. Looking forward to more Netty optimizations.
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-262) Netty optimization to handle requests
locally whenever possible
Posted by "Eugene Koontz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eugene Koontz updated GIRAPH-262:
---------------------------------
Description:
Adds optimization to handle requests locally when possible rather than go over the network
(Original description follows below: see related JIRAs for each item besides the above)
* Makes netty the default instead of HadoopRPC
* Added optimization to handle requests locally when possible rather than go over the network
* Added TimedLogger to print only within a given time period
* Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
* Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
* Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
was:
* Makes netty the default instead of HadoopRPC
* Added optimization to handle requests locally when possible rather than go over the network
* Added TimedLogger to print only within a given time period
* Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
* Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
* Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
> Netty optimization to handle requests locally whenever possible
> ---------------------------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.2.patch, GIRAPH-262.3.patch, GIRAPH-262.4.patch, GIRAPH-262.5.patch, GIRAPH-262.patch
>
>
> Adds optimization to handle requests locally when possible rather than go over the network
> (Original description follows below: see related JIRAs for each item besides the above)
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira