You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hama.apache.org by 步青云 <ma...@qq.com> on 2014/01/14 10:21:36 UTC
Some Question of Hama Program on Large Dataset.
Hello,
I've got some problem about Hama program on large dataset.My hama cluster has 3 computers.One is bspmaster,other 2 computers is groomserver.When the graph has one million nodes,the pagerank running will stop at "Current Superstep Number:0".The error information is as follows:
attempt_201401141543_0015_000000_0: 14/01/14 17:07:55 ERROR bsp.BSPTask: Shutting down ping service.
attempt_201401141543_0015_000000_0: 14/01/14 17:07:55 FATAL bsp.GroomServer: Error running child
attempt_201401141543_0015_000000_0: java.lang.OutOfMemoryError: Java heap space
attempt_201401141543_0015_000000_0: at java.util.Arrays.copyOf(Arrays.java:2271)
attempt_201401141543_0015_000000_0: at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
attempt_201401141543_0015_000000_0: at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
attempt_201401141543_0015_000000_0: at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
attempt_201401141543_0015_000000_0: at java.io.DataOutputStream.write(DataOutputStream.java:107)
Should I add the number of computers of cluster to solve this problem?Any other solution?
And I also want to know how many peers a groomserver can run?Is a peer correspond to a node in graph mode?If so,I have one million of nodes in a graph and if one groomserver run 500 peers,so I need 2000 groomservers correspond to 2000 computers.In my opinion,it's not convenient.
By the way,what's a appropriate value of "bsp.tasks.maxinum" in Hama conf ?Is the number of computer of cluster a appropriate value?
Thanks!
Best wishes.
Re： Some Question of Hama Program on Large Dataset.
Posted by 步青云 <ma...@qq.com>.
Using OffHeapVerticesInfo may be a good solution.But now I will try to use the option 'bsp.child.java.opts' at first.Because I think the graph with 50,000 nodes should not lead to 'heap space'.Each groomserver has at least 2G RAM.It's enough to load all vertices in RAM.
In my view,for the larger graph,using OffHeapVerticesInfo should be great solution.Is there any example of using OffHeapVerticesInfo?
Thanks .
Best wishes.
------------------ 原始邮件 ------------------
发件人: "Tommaso Teofili";<to...@gmail.com>;
发送时间: 2014年1月14日(星期二) 晚上9:24
收件人: "user"<us...@hama.apache.org>;
主题: Re: Some Question of Hama Program on Large Dataset.
You may also use OffHeapVerticesInfo [1] to store vertices off heap and
avoid consuming too much memory for that.
Regards,
Tommaso
[1] :
https://svn.apache.org/repos/asf/hama/trunk/graph/src/main/java/org/apache/hama/graph/OffHeapVerticesInfo.java
2014/1/14 Chia-Hung Lin <cl...@googlemail.com>
> For OOM, memory related option `bsp.child.java.opts' can be added to
> hama-site.xml for increasing memory size; or consider reducing memory
> footprint.
>
> Based on the setting at hand, max tasks, i.e. peer, can be configured
> via `bsp.tasks.maximum' with default set to 3.
>
> IIRC, peer process is something like child process in original
> mapreduce where task tracker will fork a child processing algorithm,
> mainly for fault isolation.
>
>
>
>
> On 14 January 2014 17:21, 步青云 <ma...@qq.com> wrote:
> > Hello,
> > I've got some problem about Hama program on large dataset.My hama
> cluster has 3 computers.One is bspmaster,other 2 computers is
> groomserver.When the graph has one million nodes,the pagerank running will
> stop at "Current Superstep Number:0".The error information is as follows:
> >
> >
> > attempt_201401141543_0015_000000_0: 14/01/14 17:07:55 ERROR bsp.BSPTask:
> Shutting down ping service.
> > attempt_201401141543_0015_000000_0: 14/01/14 17:07:55 FATAL
> bsp.GroomServer: Error running child
> > attempt_201401141543_0015_000000_0: java.lang.OutOfMemoryError: Java
> heap space
> > attempt_201401141543_0015_000000_0: at
> java.util.Arrays.copyOf(Arrays.java:2271)
> > attempt_201401141543_0015_000000_0: at
> java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> > attempt_201401141543_0015_000000_0: at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> > attempt_201401141543_0015_000000_0: at
> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> > attempt_201401141543_0015_000000_0: at
> java.io.DataOutputStream.write(DataOutputStream.java:107)
> > Should I add the number of computers of cluster to solve this
> problem?Any other solution?
> >
> > And I also want to know how many peers a groomserver can run?Is a
> peer correspond to a node in graph mode?If so,I have one million of nodes
> in a graph and if one groomserver run 500 peers,so I need 2000 groomservers
> correspond to 2000 computers.In my opinion,it's not convenient.
> > By the way,what's a appropriate value of "bsp.tasks.maxinum" in
> Hama conf ?Is the number of computer of cluster a appropriate value?
> > Thanks!
> > Best wishes.
>
Re: Some Question of Hama Program on Large Dataset.
Posted by Tommaso Teofili <to...@gmail.com>.
You may also use OffHeapVerticesInfo [1] to store vertices off heap and
avoid consuming too much memory for that.
Regards,
Tommaso
[1] :
https://svn.apache.org/repos/asf/hama/trunk/graph/src/main/java/org/apache/hama/graph/OffHeapVerticesInfo.java
2014/1/14 Chia-Hung Lin <cl...@googlemail.com>
> For OOM, memory related option `bsp.child.java.opts' can be added to
> hama-site.xml for increasing memory size; or consider reducing memory
> footprint.
>
> Based on the setting at hand, max tasks, i.e. peer, can be configured
> via `bsp.tasks.maximum' with default set to 3.
>
> IIRC, peer process is something like child process in original
> mapreduce where task tracker will fork a child processing algorithm,
> mainly for fault isolation.
>
>
>
>
> On 14 January 2014 17:21, 步青云 <ma...@qq.com> wrote:
> > Hello,
> > I've got some problem about Hama program on large dataset.My hama
> cluster has 3 computers.One is bspmaster,other 2 computers is
> groomserver.When the graph has one million nodes,the pagerank running will
> stop at "Current Superstep Number:0".The error information is as follows:
> >
> >
> > attempt_201401141543_0015_000000_0: 14/01/14 17:07:55 ERROR bsp.BSPTask:
> Shutting down ping service.
> > attempt_201401141543_0015_000000_0: 14/01/14 17:07:55 FATAL
> bsp.GroomServer: Error running child
> > attempt_201401141543_0015_000000_0: java.lang.OutOfMemoryError: Java
> heap space
> > attempt_201401141543_0015_000000_0: at
> java.util.Arrays.copyOf(Arrays.java:2271)
> > attempt_201401141543_0015_000000_0: at
> java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> > attempt_201401141543_0015_000000_0: at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> > attempt_201401141543_0015_000000_0: at
> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> > attempt_201401141543_0015_000000_0: at
> java.io.DataOutputStream.write(DataOutputStream.java:107)
> > Should I add the number of computers of cluster to solve this
> problem?Any other solution?
> >
> > And I also want to know how many peers a groomserver can run?Is a
> peer correspond to a node in graph mode?If so,I have one million of nodes
> in a graph and if one groomserver run 500 peers,so I need 2000 groomservers
> correspond to 2000 computers.In my opinion,it's not convenient.
> > By the way,what's a appropriate value of "bsp.tasks.maxinum" in
> Hama conf ?Is the number of computer of cluster a appropriate value?
> > Thanks!
> > Best wishes.
>
Re： Some Question of Hama Program on Large Dataset.
Posted by 步青云 <ma...@qq.com>.
Thank you.
Maybe the option 'bsp.child.java.opts' is a good solution for my current problem.
My hama cluster has 3 computers.The bspmaster has 4G RAM.One of two groomservers has 2G RAM and the other has 4G RAM.So,what's the appropriate value of the option 'bsp.child.java.opts'? Now the default value of 'bsp.child.java.opts' is '-Xmx1024m'.But I don't know the meaning of '-Xmx1024m'.
As to max tasks,I know how to set the value of `bsp.tasks.maximum' .But I don't know which the appropriate value of `bsp.tasks.maximum' is .I think the value should be the number of my groomservers.So each groomserver would have one task.Is that right?
Thanks again.It's really a great help for me.
------------------ 原始邮件 ------------------
发件人: "Chia-Hung Lin";<cl...@googlemail.com>;
发送时间: 2014年1月14日(星期二) 晚上9:15
收件人: "user"<us...@hama.apache.org>;
主题: Re: Some Question of Hama Program on Large Dataset.
For OOM, memory related option `bsp.child.java.opts' can be added to
hama-site.xml for increasing memory size; or consider reducing memory
footprint.
Based on the setting at hand, max tasks, i.e. peer, can be configured
via `bsp.tasks.maximum' with default set to 3.
IIRC, peer process is something like child process in original
mapreduce where task tracker will fork a child processing algorithm,
mainly for fault isolation.
On 14 January 2014 17:21, 步青云 <ma...@qq.com> wrote:
> Hello,
> I've got some problem about Hama program on large dataset.My hama cluster has 3 computers.One is bspmaster,other 2 computers is groomserver.When the graph has one million nodes,the pagerank running will stop at "Current Superstep Number:0".The error information is as follows:
>
>
> attempt_201401141543_0015_000000_0: 14/01/14 17:07:55 ERROR bsp.BSPTask: Shutting down ping service.
> attempt_201401141543_0015_000000_0: 14/01/14 17:07:55 FATAL bsp.GroomServer: Error running child
> attempt_201401141543_0015_000000_0: java.lang.OutOfMemoryError: Java heap space
> attempt_201401141543_0015_000000_0: at java.util.Arrays.copyOf(Arrays.java:2271)
> attempt_201401141543_0015_000000_0: at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> attempt_201401141543_0015_000000_0: at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> attempt_201401141543_0015_000000_0: at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> attempt_201401141543_0015_000000_0: at java.io.DataOutputStream.write(DataOutputStream.java:107)
> Should I add the number of computers of cluster to solve this problem?Any other solution?
>
> And I also want to know how many peers a groomserver can run?Is a peer correspond to a node in graph mode?If so,I have one million of nodes in a graph and if one groomserver run 500 peers,so I need 2000 groomservers correspond to 2000 computers.In my opinion,it's not convenient.
> By the way,what's a appropriate value of "bsp.tasks.maxinum" in Hama conf ?Is the number of computer of cluster a appropriate value?
> Thanks!
> Best wishes.
.
Re: Some Question of Hama Program on Large Dataset.
Posted by Chia-Hung Lin <cl...@googlemail.com>.
For OOM, memory related option `bsp.child.java.opts' can be added to
hama-site.xml for increasing memory size; or consider reducing memory
footprint.
Based on the setting at hand, max tasks, i.e. peer, can be configured
via `bsp.tasks.maximum' with default set to 3.
IIRC, peer process is something like child process in original
mapreduce where task tracker will fork a child processing algorithm,
mainly for fault isolation.
On 14 January 2014 17:21, 步青云 <ma...@qq.com> wrote:
> Hello,
> I've got some problem about Hama program on large dataset.My hama cluster has 3 computers.One is bspmaster,other 2 computers is groomserver.When the graph has one million nodes,the pagerank running will stop at "Current Superstep Number:0".The error information is as follows:
>
>
> attempt_201401141543_0015_000000_0: 14/01/14 17:07:55 ERROR bsp.BSPTask: Shutting down ping service.
> attempt_201401141543_0015_000000_0: 14/01/14 17:07:55 FATAL bsp.GroomServer: Error running child
> attempt_201401141543_0015_000000_0: java.lang.OutOfMemoryError: Java heap space
> attempt_201401141543_0015_000000_0: at java.util.Arrays.copyOf(Arrays.java:2271)
> attempt_201401141543_0015_000000_0: at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> attempt_201401141543_0015_000000_0: at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> attempt_201401141543_0015_000000_0: at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> attempt_201401141543_0015_000000_0: at java.io.DataOutputStream.write(DataOutputStream.java:107)
> Should I add the number of computers of cluster to solve this problem?Any other solution?
>
> And I also want to know how many peers a groomserver can run?Is a peer correspond to a node in graph mode?If so,I have one million of nodes in a graph and if one groomserver run 500 peers,so I need 2000 groomservers correspond to 2000 computers.In my opinion,it's not convenient.
> By the way,what's a appropriate value of "bsp.tasks.maxinum" in Hama conf ?Is the number of computer of cluster a appropriate value?
> Thanks!
> Best wishes.