You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Miklos Erdelyi (JIRA)" <ji...@apache.org> on 2011/04/18 17:47:05 UTC
[jira] [Created] (HAMA-380) Send messages in batches to reduce RPC
overhead.
Send messages in batches to reduce RPC overhead.
------------------------------------------------
Key: HAMA-380
URL: https://issues.apache.org/jira/browse/HAMA-380
Project: Hama
Issue Type: Improvement
Components: bsp
Affects Versions: 0.3.0
Reporter: Miklos Erdelyi
Fix For: 0.3.0
When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by Thomas Jungblut <th...@googlemail.com>.
No problem :)
2011/4/21 Edward J. Yoon <ed...@apache.org>
> Oh, my mistake.
>
> I've just committed your patch. Thanks thomas!
>
> On Thu, Apr 21, 2011 at 5:25 PM, Edward J. Yoon (JIRA) <ji...@apache.org>
> wrote:
> >
> > [
> https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022684#comment-13022684]
> >
> > Edward J. Yoon commented on HAMA-380:
> > -------------------------------------
> >
> > Please update your source code to latest TRUNK. Miklos Erdelyi's patch is
> already committed.
> >
> >
> >
> >> Send messages in batches to reduce RPC overhead.
> >> ------------------------------------------------
> >>
> >> Key: HAMA-380
> >> URL: https://issues.apache.org/jira/browse/HAMA-380
> >> Project: Hama
> >> Issue Type: Improvement
> >> Components: bsp
> >> Affects Versions: 0.3.0
> >> Reporter: Miklos Erdelyi
> >> Assignee: Miklos Erdelyi
> >> Fix For: 0.3.0
> >>
> >> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch,
> bspmsgbundle.v2.patch
> >>
> >> Original Estimate: 4h
> >> Remaining Estimate: 4h
> >>
> >> When profiling graph computations running on HAMA it turned out that
> huge amount of time is spent in RPC calls.
> >> To reduce this overhead I propose extending the GroomServer's API with a
> put(BSPMessageBundle) method so that a group of messages can be transferred
> from the caller to the callee at once. Also, the sync() operation should be
> improved by sending messages in batches rather than one-by-one.
> >> Using BSPMessageBundle makes it easier to add support for compression of
> messages later.
> >
> > --
> > This message is automatically generated by JIRA.
> > For more information on JIRA, see:
> http://www.atlassian.com/software/jira
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> http://blog.udanax.org
> http://twitter.com/eddieyoon
>
--
Thomas Jungblut
Berlin
mobile: 0170-3081070
business: thomas.jungblut@testberichte.de
private: thomas.jungblut@gmail.com
Re: [jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Edward J. Yoon" <ed...@apache.org>.
Oh, my mistake.
I've just committed your patch. Thanks thomas!
On Thu, Apr 21, 2011 at 5:25 PM, Edward J. Yoon (JIRA) <ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022684#comment-13022684 ]
>
> Edward J. Yoon commented on HAMA-380:
> -------------------------------------
>
> Please update your source code to latest TRUNK. Miklos Erdelyi's patch is already committed.
>
>
>
>> Send messages in batches to reduce RPC overhead.
>> ------------------------------------------------
>>
>> Key: HAMA-380
>> URL: https://issues.apache.org/jira/browse/HAMA-380
>> Project: Hama
>> Issue Type: Improvement
>> Components: bsp
>> Affects Versions: 0.3.0
>> Reporter: Miklos Erdelyi
>> Assignee: Miklos Erdelyi
>> Fix For: 0.3.0
>>
>> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>>
>> Original Estimate: 4h
>> Remaining Estimate: 4h
>>
>> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
>> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
>> Using BSPMessageBundle makes it easier to add support for compression of messages later.
>
> --
> This message is automatically generated by JIRA.
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
--
Best Regards, Edward J. Yoon
http://blog.udanax.org
http://twitter.com/eddieyoon
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022684#comment-13022684 ]
Edward J. Yoon commented on HAMA-380:
-------------------------------------
Please update your source code to latest TRUNK. Miklos Erdelyi's patch is already committed.
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-380) Send messages in batches to reduce RPC
overhead.
Posted by "Miklos Erdelyi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Miklos Erdelyi updated HAMA-380:
--------------------------------
Description:
When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
Using BSPMessageBundle makes it easier to add support for compression of messages later.
was:
When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Fix For: 0.3.0
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021959#comment-13021959 ]
Edward J. Yoon commented on HAMA-380:
-------------------------------------
Hi Thomas, I wanted to see your patch but it does not work against trunk right now. Could you please upload again?
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021984#comment-13021984 ]
Thomas Jungblut commented on HAMA-380:
--------------------------------------
Why? Built against the last revision where you checked in this patch here.
Don't know if an instanceof call will prevent the warning, but I would disadvice from using it. It's lame as hell :D
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-380) Send messages in batches to reduce RPC
overhead.
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-380:
--------------------------------
Resolution: Fixed
Assignee: Miklos Erdelyi
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Miklos!
P.S., Please use the formatter.
http://incubator.apache.org/hama/developers.html
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-380) Send messages in batches to reduce RPC
overhead.
Posted by "Miklos Erdelyi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Miklos Erdelyi updated HAMA-380:
--------------------------------
Attachment: bspmsgbundle.patch
Patch adding BSPMessageBundle.
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: bspmsgbundle.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027906#comment-13027906 ]
Hudson commented on HAMA-380:
-----------------------------
Integrated in Hama-Nightly #189 (See [https://builds.apache.org/hudson/job/Hama-Nightly/189/])
HAMA-380 is a new feature.
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-380) Send messages in batches to reduce RPC
overhead.
Posted by "Miklos Erdelyi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Miklos Erdelyi updated HAMA-380:
--------------------------------
Attachment: bspmsgbundle.v2.patch
Attached patch which includes unittests for BSPMessageBundle too.
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028312#comment-13028312 ]
Hudson commented on HAMA-380:
-----------------------------
Integrated in Hama-Patch #327 (See [https://builds.apache.org/hudson/job/Hama-Patch/327/])
HAMA-380 is a new feature.
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021078#comment-13021078 ]
Hudson commented on HAMA-380:
-----------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12476620/bspmsgbundle.patch
against trunk revision 1090506.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs -1. The patch appears to cause Findbugs to fail.
core tests +1. The patch passed core unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hama-Patch/321/testReport/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hama-Patch/321/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hama-Patch/321/console
This message is automatically generated.
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021964#comment-13021964 ]
Edward J. Yoon commented on HAMA-380:
-------------------------------------
Plus, is there a way to avoid unchecked cast warning?
Below code seems OK.
{code}
Class<?> clazz = null;
...
BSPMessage msg = (BSPMessage) ReflectionUtils.newInstance(clazz, null);
{code}
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021542#comment-13021542 ]
Thomas Jungblut commented on HAMA-380:
--------------------------------------
The order is not the problem, the problem was to support multiple messages in a bundle.
No one actually should write the size and vertices and then expect to come elements in that ordering. We just should support this, because this was actually working before the patch. There are plenty of situations where this could be useful, but my example wasn't well picked ;)
Never the less I agree with your design POV!
Nice work! +1
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Miklos Erdelyi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021537#comment-13021537 ]
Miklos Erdelyi commented on HAMA-380:
-------------------------------------
* You're right that caching constructors minimizes the time needed to instantiate classes. Thanks for adding this!
* However, in the BSP model there is no guarantee to the order of received messages so you cannot make assumptions like 'receive 1 Foo class then receive 3 Bar classes'. That is why I originally didn't deal with the situation where mixed classes might be added to a BSPMessageBundle.
* From a design point of view, you can always embed different message classes into one container message class so the simpler solution (i.e., to support buffering instances of only one class in a bundle) still works.
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021130#comment-13021130 ]
Thomas Jungblut commented on HAMA-380:
--------------------------------------
I like that +1 and reviewed the patch.
Do you have a comparision?
I believe that using reflection here without caching can be a bottleneck too. Using the void constructor (searching it once) and cache it is a lot better.
{noformat}
try {
clazz = (Class<? extends BSPMessage>) Class.forName(className);
} catch (ClassNotFoundException ex) {
throw new IOException(ex);
}
// Deserialize messages.
messages = new ArrayList<BSPMessage>(numMessages);
for (int i = 0; i < numMessages; ++i) {
try {
// Instantiate new message and deserialize it.
BSPMessage newMessage = clazz.newInstance();
{noformat}
Have a look at org.apache.hadoop.util.ReflectionUtils and check out: newInstance(Class<T> theClass, Configuration conf).
Maybe we should sort the messages by their class, this will reduce traffic with sending the name of the class for each message.
Otherwise nice work ;)
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-380) Send messages in batches to reduce RPC
overhead.
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-380:
---------------------------------
Attachment: HAMA-380-v2.patch
attaching it as: HAMA-380-v2.patch
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021534#comment-13021534 ]
Thomas Jungblut commented on HAMA-380:
--------------------------------------
Hi,
actually I have a few things:
* You're just sending the first class of the element in messagequeue. What if a user is using several message classes and want to sync them? E.g. sending the size of an message as an integermessage and afterwards a list of vertices as a vertexmessage?
{noformat}
BSPMessage element = messages.get(0);
Class<? extends BSPMessage> clazz = element.getClass();
{noformat}
* Lot's of overhead with searching classes and instantiating.
I targeted it and added a class cache and structured the data a bit more to fit with the multiple classes.
It's ~ 20% faster than the version you've provided.
Would be great to see a benchmark for it. I mean randbench is not meaningful at all, because of the random component.
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-380) Send messages in batches to reduce RPC
overhead.
Posted by "Miklos Erdelyi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Miklos Erdelyi updated HAMA-380:
--------------------------------
Comment: was deleted
(was: Patch adding BSPMessageBundle.)
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: bspmsgbundle.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-380) Send messages in batches to reduce RPC
overhead.
Posted by "Miklos Erdelyi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Miklos Erdelyi updated HAMA-380:
--------------------------------
Status: Patch Available (was: Open)
Patch adding BSPMessageBundle.
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Fix For: 0.3.0
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021367#comment-13021367 ]
Edward J. Yoon commented on HAMA-380:
-------------------------------------
Looks great patch! +1
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022260#comment-13022260 ]
Thomas Jungblut commented on HAMA-380:
--------------------------------------
Sorry for my late reply. But I'm on revision 1094852 and I actually build this patch: HAMA-380-v2.patch against it.
So for me it is working.
What's the problem you're facing?
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021986#comment-13021986 ]
Thomas Jungblut commented on HAMA-380:
--------------------------------------
Oh noes! Yesterday it worked with the current trunk :D
I'll do it later, I'm currently in a lecture and the wlan does not permit the svn port :/
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-380) Send messages in batches to reduce
RPC overhead.
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021985#comment-13021985 ]
Edward J. Yoon commented on HAMA-380:
-------------------------------------
"svn update"!
> Send messages in batches to reduce RPC overhead.
> ------------------------------------------------
>
> Key: HAMA-380
> URL: https://issues.apache.org/jira/browse/HAMA-380
> Project: Hama
> Issue Type: Improvement
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Miklos Erdelyi
> Assignee: Miklos Erdelyi
> Fix For: 0.3.0
>
> Attachments: HAMA-380-v2.patch, bspmsgbundle.patch, bspmsgbundle.v2.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> When profiling graph computations running on HAMA it turned out that huge amount of time is spent in RPC calls.
> To reduce this overhead I propose extending the GroomServer's API with a put(BSPMessageBundle) method so that a group of messages can be transferred from the caller to the callee at once. Also, the sync() operation should be improved by sending messages in batches rather than one-by-one.
> Using BSPMessageBundle makes it easier to add support for compression of messages later.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira