Run giraph on Hadoop yarn 2.6.0

[Flavian Jacquot <tk...@gmail.com>]

Hello, I'm trying to use Giraph on hadoop 2.6.0 with yarn.

I've managed to build it by removing STATIC_SASL_SYMBOL in <munge.symbols>
in the yarn profile. with the command :
sudo mvn -Phadoop_yarn -Dhadoop.version=2.6.0 -DskipTests package

Then i've setup a single node cluster with hadoop 2.6.0 and I'm able to
perform the basic wordcount example.
I've copied the jar built into the directory share/hadoop/mapreduce to
remove the GiraphMaster not found execption.

Now I'm trying to run the SimpleShortestPaths example with the command :

$HADOOP_PREFIX/bin/hadoop jar
/usr/local/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.6.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip input_tiny2.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/hduser/output/shortestpaths445 -w 1 -ca
giraph.SplitMasterWorker=false -yj
giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.6.0-jar-with-dependencies.jar

The data used is the tiny graph on the giraph quick start page and the job
should takes minutes to complete but it doesn't.

The full log is at http://pastebin.com/CW1FnWz2

It's possible to make giraph work on hadoop 2.6.0 or do I have to use an
earlier version ? If so which one would you suggest ?

Re: Run giraph on Hadoop yarn 2.6.0

[Alessandro Negro <al...@yahoo.it>]

Hi Flavian,
do you try follow all this thread:

http://mail-archives.apache.org/mod_mbox/giraph-user/201411.mbox/%3cCA+BectmBb2x3MXsn_e0G_4iUWtwLHVWYYteu9HDVt+h7VQUMgg@mail.gmail.com%3e

In any case I’ll test it as soon as I can against the new version.

Cheers,
Alessandro


Il giorno 02/dic/2014, alle ore 11:09, Flavian Jacquot <tk...@gmail.com> ha scritto:

> Hello, I'm trying to use Giraph on hadoop 2.6.0 with yarn.
> 
> I've managed to build it by removing STATIC_SASL_SYMBOL in <munge.symbols> in the yarn profile. with the command :
> sudo mvn -Phadoop_yarn -Dhadoop.version=2.6.0 -DskipTests package
> 
> Then i've setup a single node cluster with hadoop 2.6.0 and I'm able to perform the basic wordcount example.
> I've copied the jar built into the directory share/hadoop/mapreduce to remove the GiraphMaster not found execption.
> 
> Now I'm trying to run the SimpleShortestPaths example with the command :
> 
> $HADOOP_PREFIX/bin/hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.6.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip input_tiny2.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths445 -w 1 -ca giraph.SplitMasterWorker=false -yj giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.6.0-jar-with-dependencies.jar
> 
> The data used is the tiny graph on the giraph quick start page and the job should takes minutes to complete but it doesn't.
> 
> The full log is at http://pastebin.com/CW1FnWz2
> 
> It's possible to make giraph work on hadoop 2.6.0 or do I have to use an earlier version ? If so which one would you suggest ?