You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@giraph.apache.org by Jyoti Yadav <ra...@gmail.com> on 2014/02/02 17:25:42 UTC

Running out of space..

Hi folks..

While executing one algorithm with enabling checkpointing feature on,system
is running out of memory. I gave following command


hadoop jar
$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner -Dmapred.job.map.memory.mb=1500
-Dgiraph.useSuperstepCounters=false -Dgiraph.useOutOfCoreMessages=true
-Dgiraph.checkpointFrequency=1
org.apache.giraph.examples.MyShortestDistance -vif
org.apache.giraph.examples.io.formats.MyShortestDistanceVertexInputFormat
-vip /user/hduser/big_input/clustering -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/hduser/sp_output537/sd_output -w 1 -mc
org.apache.giraph.examples.MyShortestDistance\$MyMasterCompute


By executing this command   /app/hadoop/tmp/dfs/name/data/current directory
is consuming lots of space,approx  90GB(or may be more than this)
This job execution is involving thousands of supersteps..If i do not enable
checkpoint feature  on then algorithm fails due to worker failure error
after 19000 around supersteps.

Pls suggest me how to deal with this issue??

Thanks
Jyoti