You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon (JIRA)" <ji...@apache.org> on 2011/03/07 11:12:59 UTC
[jira] Created: (HAMA-358) Evaluation of communication protocol
performance
Evaluation of communication protocol performance
------------------------------------------------
Key: HAMA-358
URL: https://issues.apache.org/jira/browse/HAMA-358
Project: Hama
Issue Type: Improvement
Components: bsp, documentation
Affects Versions: 0.2.0
Reporter: Edward J. Yoon
Fix For: 0.3.0
The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050813#comment-13050813 ]
Bill Graham commented on HAMA-358:
----------------------------------
Could Avro help here? It doesn't require compile-time schema knowledge (you don't need to generate code stubs) so the user could define an Avro schema and then work with that at run time.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Zubair Nabi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015987#comment-13015987 ]
Zubair Nabi commented on HAMA-358:
----------------------------------
Perfect. Thanks. I'll start writing my proposal.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Labels: gsoc, gsoc2011, mentor
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon reassigned HAMA-358:
-----------------------------------
Assignee: Edward J. Yoon
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059229#comment-13059229 ]
Thomas Jungblut commented on HAMA-358:
--------------------------------------
I asked myself if we need to strictly stick with RPC for BSP Communication.
What about just setting up a SequenceFile and sending it across the network using plain old streams?
We can later add asynchronous communication during computational phases if the SequenceFile reached it's blocksize...
What do you think of it?
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057641#comment-13057641 ]
Edward J. Yoon commented on HAMA-358:
-------------------------------------
{code}
eddieyoon Edward J. Yoon
@
@tjungblut By the way, serializable protobuf seems meaningless by BSPMessageBundle. What do you think?
3 minutes ago
in reply to @eddieyoon ↑
@tjungblut
Thomas Jungblut
@eddieyoon I agree, we should look into avro
{code}
Copy&paste chat log.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015669#comment-13015669 ]
Edward J. Yoon commented on HAMA-358:
-------------------------------------
Hadoop RPC, Thrift, and Google Protobuf are enough. and Your plan looks pretty good. :)
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Labels: gsoc, gsoc2011, mentor
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011887#comment-13011887 ]
Edward J. Yoon commented on HAMA-358:
-------------------------------------
I'm mentoring this project.
* name: Edward J. Yoon
* link_id: edwardyoon
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Labels: gsoc, gsoc2011, mentor
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HAMA-358) Evaluation of Hama BSP
communication protocol performance
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050509#comment-13050509 ]
Thomas Jungblut edited comment on HAMA-358 at 6/16/11 10:17 PM:
----------------------------------------------------------------
How is it going? Do you need help?
I've practiced a bit with ProtoBuf.
It seems to be really cool and fast (or at least small).
BUT I have really problems with the question: how we are (or better we could be) able to integrate it.
Let's summarize our current state:
We have an abstract class BSPMessage which leaves the types of tags and data to the concrete implementations.
This gets using Writable into Hadoop's RPC mechanism and gets serialized and deserialized.
What I'm wondering now is how we can improve this using ProtoBuf.
ProtoBuf needs a "*.proto" file that needs to be compiled to a specific model *.java file. In this file you are declaring what the message needs, in our case this is not known at compile time (for example a user implements a custom vertex that contains distances or something like that). So we have to leave the serialization up to the user.
The question is how could we doing this?
Here some thoughts (don't take this too serious, just some brainstorming):
There are two options:
* A generic ".proto" model that takes just two Strings for a tag and data
* We leave the compiling and implementing of the protos to the user
The first is ultra simple and we don't have to worry about anything the user will submit, since you can serialize everything to strings.
But I think we are going to "ruin" the optimizations that COULD have been made if the type was known.
The second option is really messy and not too user friendly (since he has to compile the whole stuff and put it into a repository at runtime that we can know the proto), but in constrast to the first option it could result in better results.
Do we have other opportunities?
was (Author: thomas.jungblut):
How is it going? Do you need help?
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050779#comment-13050779 ]
Thomas Jungblut commented on HAMA-358:
--------------------------------------
Yap, I had an older index application that runs on a server, wanted to replace RMI with this. But I gave up after two hours, it seems to be written for python primarily.
No :) Was just interested in what your results might be.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050509#comment-13050509 ]
Thomas Jungblut commented on HAMA-358:
--------------------------------------
How is it going? Do you need help?
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-358:
--------------------------------
Affects Version/s: (was: 0.2.0)
0.3.0
Fix Version/s: (was: 0.3.0)
0.4.0
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036712#comment-13036712 ]
Edward J. Yoon commented on HAMA-358:
-------------------------------------
I'm going to evaluate the performance of Google's protobuf.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "ChiaHung Lin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026930#comment-13026930 ]
ChiaHung Lin commented on HAMA-358:
-----------------------------------
If that's possible, I would like other framework to be tested as well such as Bert (bert-rpc.org). But the main concerns is the issues related to asynchronous (or message passing) communication and the data serialization mechanism.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Labels: gsoc, gsoc2011, mentor
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005459#comment-13005459 ]
Edward J. Yoon commented on HAMA-358:
-------------------------------------
Quick guide: http://wiki.apache.org/hama/GSoC2011
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Labels: gsoc, gsoc2011, mentor
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121339#comment-13121339 ]
Thomas Jungblut commented on HAMA-358:
--------------------------------------
We can take a look at ProtoStuff:
http://code.google.com/p/protostuff/
I've seen that directmemory use it for serialization. Maybe it can be interesting for us too.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Task
> Components: bsp, documentation
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050776#comment-13050776 ]
Edward J. Yoon commented on HAMA-358:
-------------------------------------
Have you seen this project? http://code.google.com/p/protobuf-socket-rpc/
I'm early stage. Do you want to work for this task?
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon reassigned HAMA-358:
-----------------------------------
Assignee: Edward J. Yoon
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Labels: gsoc, gsoc2011, mentor
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158686#comment-13158686 ]
Thomas Jungblut commented on HAMA-358:
--------------------------------------
Here is avro:
https://github.com/thomasjungblut/hama-avro/blob/master/src/de/jungblut/avro/AvroMessageServiceImpl.java
I think the overhead might be larger than the RPC of Hadoop, but it's just a guess.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Task
> Components: bsp, documentation
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Thomas Jungblut (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut resolved HAMA-358.
----------------------------------
Resolution: Won't Fix
Let's close this, we have benchmarks of Hadoop RPC vs. Avro here http://wiki.apache.org/hama/Benchmarks#Compare_between_Hadoop_RPC_and_Avro
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Task
> Components: bsp, documentation
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145063#comment-13145063 ]
Thomas Jungblut commented on HAMA-358:
--------------------------------------
I made a bit research with Avro and Protobuf.
https://github.com/thomasjungblut/hama-avro
Once HAMA-461 is done we can have a look and benchmark it.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Task
> Components: bsp, documentation
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-358:
--------------------------------
Summary: Evaluation of Hama BSP communication protocol performance (was: Evaluation of communication protocol performance)
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Labels: gsoc, gsoc2011, mentor
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HAMA-358) Evaluation of communication protocol
performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-358:
--------------------------------
Labels: gsoc gsoc2011 (was: )
> Evaluation of communication protocol performance
> ------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Labels: gsoc, gsoc2011
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050828#comment-13050828 ]
Edward J. Yoon commented on HAMA-358:
-------------------------------------
I have to look at Avro, too.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.4.0
>
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Zubair Nabi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015383#comment-13015383 ]
Zubair Nabi commented on HAMA-358:
----------------------------------
Hi,
I would like more information about this project. I have extensive background in both Hadoop, Thrift and distributed systems in general. My question is that how many frameworks do you plan on evaluating? Because the plane of RPC frameworks is pretty large. Do you have an upper bound in mind?
Secondly, as I understand, these are going to be the steps of the project:
1) Coming up with a list of features that are required for Hama BSP communication.
2) Short-listing RPC frameworks based on the exhaustive list of features.
3) Evaluation of these short-listed frameworks in a real cluster environment.
4) Presenting this evaluation in a formal manner with graphs - in the form of a report.
5) Recommending a final framework based on the evaluation.
Am I missing anything?
Thanks,
Zubair
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Labels: gsoc, gsoc2011, mentor
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-358:
--------------------------------
Affects Version/s: (was: 0.3.0)
Fix Version/s: (was: 0.4.0)
Issue Type: Task (was: Improvement)
I just changed the type of this issue to research task.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Task
> Components: bsp, documentation
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Original Estimate: 1,008h
> Remaining Estimate: 1,008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon reassigned HAMA-358:
-----------------------------------
Assignee: (was: Edward J. Yoon)
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Labels: gsoc, gsoc2011, mentor
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025104#comment-13025104 ]
Thomas Jungblut commented on HAMA-358:
--------------------------------------
So Zubair, you wasn't picked. Are you although going to evaluate it?
I recently recognized that Hadoop RPC wasn't really fast. It is used at startup when the datanodes sending the blocks to the namenode. It takes 20 minutes for arround 2mio files on 6 nodes. That is really slow.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Labels: gsoc, gsoc2011, mentor
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004955#comment-13004955 ]
Edward J. Yoon commented on HAMA-358:
-------------------------------------
FYI, I can provide some testbed servers (200~500 nodes) If needed.
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Labels: gsoc, gsoc2011, mentor
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HAMA-358) Evaluation of Hama BSP
communication protocol performance
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036864#comment-13036864 ]
Thomas Jungblut edited comment on HAMA-358 at 5/20/11 3:10 PM:
---------------------------------------------------------------
I'm excited about the results ;D
was (Author: thomas.jungblut):
I'm excited on the results ;D
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-358) Evaluation of Hama BSP communication
protocol performance
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036864#comment-13036864 ]
Thomas Jungblut commented on HAMA-358:
--------------------------------------
I'm excited on the results ;D
> Evaluation of Hama BSP communication protocol performance
> ---------------------------------------------------------
>
> Key: HAMA-358
> URL: https://issues.apache.org/jira/browse/HAMA-358
> Project: Hama
> Issue Type: Improvement
> Components: bsp, documentation
> Affects Versions: 0.2.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.3.0
>
> Original Estimate: 1008h
> Remaining Estimate: 1008h
>
> The goal of this project is performance evaluation of RPC frameworks (e.g., Hadoop RPC, Thrift, Google Protobuf, ..., etc) to figure out which is the best solution for Hama BSP communication. Currently Hama is using Hadoop RPC to communicate and transfer messages between BSP workers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira