You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hama.apache.org by Paweł Brach <br...@gmail.com> on 2011/02/02 10:25:37 UTC

hama - general questions

Hi,

I'm trying to learn Hama. I've set Hadoop cluster on 3 machines and Hama on
it (also 3 nodes). Everything works I think, I'm able to run example codes
like Pi and Test (provided with Hama).
Now I want to write own code - I'm trying to create a graph and calclate
something on it using Hama. To do that I need to understand better how it
works.
I want to implement for example Page Rank algorithm (in the naive way).
If we consider typical Hama job, there is a main method where we set a
cluster and set jobs for nodes.
I want to create random graph in the main method, divide set of vertices
into 3 groups (I have 3 Hama nodes) and in every Hama node process 1/3
vertices.

The Hama node code skeleton (one superstep):
- for each vertex: send current PR value to its neighbours (i.e. Hama node
which is processing this neighbour)
- sync
- for each vertex: receive messages and update PR value

Questions:
- As I mentioned before in the main method in the Hama job, I would like to
create random graph and set 1/3 vertices to every Hama node. The questions
is how to make vertices accessible for Hama nodes (in bsp method). Probably
I need to put all vertices in the HDFS and get them in every Hama node from
HDFS, but how to do that? How does Hama provide access to HDFS ?
- On the bottom of Hama is HDFS, I have cluster with 3 machines, on every
nodes there are DataNode from Hadoop and Hama node. When some Hama node want
to get something from HDFS it could communicate with DataNode which is
located on different machine. How to force to use DataNode located on the
same machine?

Thanks for help,
Cheers,
Pawel

Re: hama - general questions

Posted by "Edward J. Yoon" <ed...@apache.org>.
Hi,

That's good plan.

> Questions:
> - As I mentioned before in the main method in the Hama job, I would like to
> create random graph and set 1/3 vertices to every Hama node. The questions
> is how to make vertices accessible for Hama nodes (in bsp method). Probably
> I need to put all vertices in the HDFS and get them in every Hama node from
> HDFS, but how to do that? How does Hama provide access to HDFS ?
> - On the bottom of Hama is HDFS, I have cluster with 3 machines, on every
> nodes there are DataNode from Hadoop and Hama node. When some Hama node want
> to get something from HDFS it could communicate with DataNode which is
> located on different machine. How to force to use DataNode located on the
> same machine?

The current version has no I/O capabilities in BSP job but you can use
the 'FileSystem' object to handles files on HDFS. For now, the simple
way is to make manually some sub-graphs for 3 machines.

On Wed, Feb 2, 2011 at 6:25 PM, Paweł Brach <br...@gmail.com> wrote:
> Hi,
>
> I'm trying to learn Hama. I've set Hadoop cluster on 3 machines and Hama on
> it (also 3 nodes). Everything works I think, I'm able to run example codes
> like Pi and Test (provided with Hama).
> Now I want to write own code - I'm trying to create a graph and calclate
> something on it using Hama. To do that I need to understand better how it
> works.
> I want to implement for example Page Rank algorithm (in the naive way).
> If we consider typical Hama job, there is a main method where we set a
> cluster and set jobs for nodes.
> I want to create random graph in the main method, divide set of vertices
> into 3 groups (I have 3 Hama nodes) and in every Hama node process 1/3
> vertices.
>
> The Hama node code skeleton (one superstep):
> - for each vertex: send current PR value to its neighbours (i.e. Hama node
> which is processing this neighbour)
> - sync
> - for each vertex: receive messages and update PR value
>
> Questions:
> - As I mentioned before in the main method in the Hama job, I would like to
> create random graph and set 1/3 vertices to every Hama node. The questions
> is how to make vertices accessible for Hama nodes (in bsp method). Probably
> I need to put all vertices in the HDFS and get them in every Hama node from
> HDFS, but how to do that? How does Hama provide access to HDFS ?
> - On the bottom of Hama is HDFS, I have cluster with 3 machines, on every
> nodes there are DataNode from Hadoop and Hama node. When some Hama node want
> to get something from HDFS it could communicate with DataNode which is
> located on different machine. How to force to use DataNode located on the
> same machine?
>
> Thanks for help,
> Cheers,
> Pawel
>



-- 
Best Regards, Edward J. Yoon
http://blog.udanax.org
http://twitter.com/eddieyoon