You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hama.apache.org by Xuwenhua <xu...@huawei.com> on 2013/12/19 10:06:26 UTC

答复: 答复: About hama performance

Hi Edward,
I only tested pagerank.

Another problem is “OutOfMem”. The test environment is the same as I mentioned in last mail.
For hama, I tested with 0.4M vertex with 100 edges using 20 task, it shows “OutOfMem”. And the local file size is about 260M.
But for giraph, I tested with 10M vertex with 10 edges using 19 task, it can run well. And the local file size is about 830M.

So do you encounter the same problem and how to avoid it?? I think the test data is not very large.

----------------------------------------------------------------------------------------------------------------------------------------------------------
Wenhua XU
Research Dept. of Software Co., Ltd
[cid:image001.jpg@01C802B8.FA7E5340] To enrich life through communication
Office: +86-755-28970151 Fax: +86-755-28975077
----------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
----------------------------------------------------------------------------------------------------------------------------------------------------------

发件人: Edward J. Yoon [mailto:edwardyoon@apache.org]
发送时间: 2013年12月13日 19:29
收件人: Xuwenhua
抄送: user@hama.apache.org; dev@hama.apache.org
主题: Re: 答复: About hama performance

> According to your message, I think this issue shall be high priority, right?? So which version and when will it be fixed and released??

Yes, it's on my todo list. I plan to fix before releasing version 0.7. BTW, have you compared the performance of SSSP or K-Means? You'll see another result.

I'm CC'ing user@h.a.o<ma...@h.a.o> and dev@h.a.o<ma...@h.a.o> lists.
On Fri, Dec 13, 2013 at 7:26 PM, Xuwenhua <xu...@huawei.com>> wrote:
Hi Edward,
Thank you for your information.

You mean I re-send this email in hama user@ list? Or you copy it to list directly??
For I do not subscribe the list now, so if need I re-send, I will send it next week. If you can directly copy it, and you can do it.

According to your message, I think this issue shall be high priority, right?? So which version and when will it be fixed and released??

----------------------------------------------------------------------------------------------------------------------------------------------------------
Wenhua XU
Research Dept. of Software Co., Ltd
[cid:image001.jpg@01C802B8.FA7E5340] To enrich life through communication
Office: +86-755-28970151<tel:%2B86-755-28970151> Fax: +86-755-28975077<tel:%2B86-755-28975077>
----------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
----------------------------------------------------------------------------------------------------------------------------------------------------------

发件人: Edward J. Yoon [mailto:edwardyoon@apache.org<ma...@apache.org>]
发送时间: 2013年12月13日 17:47
收件人: Xuwenhua
主题: Re: About hama performance

Hi Xuwenhua,

> From the test report, we can see that hama cost twice time.

I guess you did correctly. Hama's Graph package is still under heavy development.

Currently Hama uses additional supersteps to broadcast aggregators, and sends messages twice (https://issues.apache.org/jira/browse/HAMA-734). It means that the performance will be improved by at least 50%.

BTW, can we move this to Hama user@ list?

On Fri, Dec 13, 2013 at 5:08 PM, Xuwenhua <xu...@huawei.com>> wrote:
Dear Edward,
This is Wenhua From Huawei. We are interested in Hama for it BSP feature.
And those days I study some hama material and do some performance tests compared with Giraph.

I used 3 servers with total core 20 as follows, and 1G for “bsp.child.java.opts”
CPU(8 core) * 1, 8G*6 master/name node，slave/data node
CPU(8 core）* 1，4G*6 slave/data node
CPU(2 core) * 2，2G*4 slave/data node
And the input format is like “vertex neighbour1 neighbour2”.

I tested from 0.1m vertex with 10 edges to 1m vertex with 10 edges.
Giraph using 19 task and Hama using 20 task, so that each core can deal with one task.
I test the pagerank, I force it to run 30 iterations,
if(this.getSuperstepCount() >= 30)
{
voteToHalt();
return;
}
And the test result is as follows.
[cid:image003.png@01CEF818.4E9CDE50]
From the test report, we can see that hama cost twice time.

So my query is that is there any problem in my experiments??
And do you have some hama performance information compared with Giraph?? Because I only see some hama’s own performance in hama website.

Looking forward to receiving your comments.

Best Regards

----------------------------------------------------------------------------------------------------------------------------------------------------------
Wenhua XU
Research Dept. of Software Co., Ltd
[cid:image001.jpg@01C802B8.FA7E5340] To enrich life through communication
Office: +86-755-28970151<tel:%2B86-755-28970151> Fax: +86-755-28975077<tel:%2B86-755-28975077>
----------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
----------------------------------------------------------------------------------------------------------------------------------------------------------

--
Best Regards, Edward J. Yoon
@eddieyoon