You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Jing Fan <fa...@gmail.com> on 2014/07/28 22:29:33 UTC
Giraph get stuck when use outOfCore option
Hi everyone,
I was running a job with outOfCore option and Giraph get stuck.
The RAM is 96G and the dataset is 20G. The server has 16 physical cores. I
was running PageRank with the following command
/usr/local/hadoop/bin/hadoop jar
/usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner -D giraph.useOutOfCoreGraph=true -D
giraph.useOutofCoreMessages=true -D giraph.maxPartitionsInMemory=50 -D
giraph.maxMessagesInMemory=500000000 -Dgiraph.zkMinSessionTimeout=9000000
-Dgiraph.zkMaxSessionTimeout=9000000 -Dgiraph.zkSessionMsecTimeout=9000000
org.apache.examples.PageRank -eif
org.apache.giraph.io.formats.IntNullTextEdgeInputFormat -eip
/user/hduser/xx.txt -w 14
It get stuck with:
14/07/28 22:05:58 INFO job.JobProgressTracker: Data from 14 workers -
Compute superstep 1: 38731088 out of 41652230 vertices computed; 182 out of
196 partitions computed; min free memory on worker 4 - 768.51MB, average
1296.96MB
and then was killed.
I check the mem, it uses nearly 100% of the memory.
Does anyone know the reason?
Thanks!
Re: Giraph get stuck when use outOfCore option
Posted by Jing Fan <fa...@gmail.com>.
Does anyone know the reason?
Thank you very much!!
On Mon, Jul 28, 2014 at 1:29 PM, Jing Fan <fa...@gmail.com> wrote:
> Hi everyone,
>
> I was running a job with outOfCore option and Giraph get stuck.
>
> The RAM is 96G and the dataset is 20G. The server has 16 physical cores.
> I was running PageRank with the following command
>
> /usr/local/hadoop/bin/hadoop jar
> /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner -D giraph.useOutOfCoreGraph=true -D
> giraph.useOutofCoreMessages=true -D giraph.maxPartitionsInMemory=50 -D
> giraph.maxMessagesInMemory=500000000 -Dgiraph.zkMinSessionTimeout=9000000
> -Dgiraph.zkMaxSessionTimeout=9000000 -Dgiraph.zkSessionMsecTimeout=9000000
> org.apache.examples.PageRank -eif
> org.apache.giraph.io.formats.IntNullTextEdgeInputFormat -eip
> /user/hduser/xx.txt -w 14
>
> It get stuck with:
>
> 14/07/28 22:05:58 INFO job.JobProgressTracker: Data from 14 workers -
> Compute superstep 1: 38731088 out of 41652230 vertices computed; 182 out of
> 196 partitions computed; min free memory on worker 4 - 768.51MB, average
> 1296.96MB
> and then was killed.
>
> I check the mem, it uses nearly 100% of the memory.
>
> Does anyone know the reason?
>
> Thanks!
>