You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Vishal Mishra <mi...@gmail.com> on 2014/03/10 04:59:32 UTC
Steps for Giraph on hadoop-2.2.0 Cluster
Hi Guys,
I tried giraph on hadoop-2.2.0 cluster a few months back without success
and moved onto other pressing issues but would really love to try out again.
Has anyone been able to run giraph successfully on a hadoop-2.2.0 cluster?
if yes, I'd really appreciate it if you can share the steps to run the
"SimplePageRankComputation" example.
If not, please share the steps, I'd give it a try and provide any required
logs for debugging.
Our setup (works for our regular hadoop/hive jobs):
- ubuntu 13.10
- 1 namenode & resourcemanager. + zookeeper
- 3 datanode & nodemanager
Thanx,
Vishal
Re: Steps for Giraph on hadoop-2.2.0 Cluster
Posted by Alexandre Fonseca <al...@gmail.com>.
This would be giraph from trunk. Giraph 1.0 is bound to have several
compatibility problems as it's very old by now (although I can't say I've
tried using it with 2.2.0).
Cheers,
Alex
On Wednesday, March 12, 2014 03:56:53 PM Rob Paul wrote:
> Hi Alex,
>
> Are you talking about Giraph 1.0 or giraph from the trunk?
> Thanks
>
> On Mon, Mar 10, 2014 at 3:11 AM, Alexandre Fonseca
>
> <al...@gmail.com> wrote:
> > Hello Vishal,
> >
> > I've run it successfully in hadoop-2.2.0 with both MR2 and Yarn
> > computations.
> >
> > For MR2, you just have to compile Giraph with the new hadoop_2 profile and
> > then it should work with no problems using the commands shown in the
> > Giraph quickstart.
> >
> > For Yarn, compile with the hadoop_yarn profile and the application
> > submission command follows the following template (replace variables with
> > your particular values):
> >
> > GIRAPH_JAR_FOLDER=/home/alex/Giraph
> > GIRAPH_JAR_NAME=giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-
> > dependencies.jar
> > GIRAPH_JAR=$GIRAPH_JAR_FOLDER/$GIRAPH_JAR_NAME
> >
> > HADOOP_CLASSPATH=$GIRAPH_JAR_FOLDER hadoop jar $GIRAPH_JAR
> > org.apache.giraph.GiraphRunner
> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> > -vip /user/alex/input/tiny_graph.txt -vof
> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> > /user/alex/output/shortestpaths -w 1 -yj $GIRAPH_JAR_NAME
> >
> > If you don't want to compile things yourself, the grafos.ml in which I'm
> > working at the moment provides, among other things, pre-compiled versions
> > of giraph straight from trunk for the major targets integrated with some
> > machine learning algorithms. You can check the download page here:
> > http://grafos.ml/Download.html
> >
> > We also have a configuration tool that should make it easy to find the
> > exact command to run different algorithms (including most of Giraph
> > examples): http://grafos.ml/configure.html
> >
> > Hopefully this should be sufficient for you to successfully run Giraph in
> > 2.2.0.
> >
> > Best regards,
> > Alex
> >
> > On Sunday, March 09, 2014 08:59:32 PM Vishal Mishra wrote:
> >> Hi Guys,
> >> I tried giraph on hadoop-2.2.0 cluster a few months back without success
> >> and moved onto other pressing issues but would really love to try out
> >> again.
> >>
> >> Has anyone been able to run giraph successfully on a hadoop-2.2.0
> >> cluster?
> >> if yes, I'd really appreciate it if you can share the steps to run the
> >> "SimplePageRankComputation" example.
> >> If not, please share the steps, I'd give it a try and provide any
> >> required
> >> logs for debugging.
> >> Our setup (works for our regular hadoop/hive jobs):
> >> - ubuntu 13.10
> >> - 1 namenode & resourcemanager. + zookeeper
> >> - 3 datanode & nodemanager
> >>
> >> Thanx,
> >> Vishal
Re: Steps for Giraph on hadoop-2.2.0 Cluster
Posted by Rob Paul <ur...@gmail.com>.
Hi Alex,
Are you talking about Giraph 1.0 or giraph from the trunk?
Thanks
On Mon, Mar 10, 2014 at 3:11 AM, Alexandre Fonseca
<al...@gmail.com> wrote:
> Hello Vishal,
>
> I've run it successfully in hadoop-2.2.0 with both MR2 and Yarn computations.
>
> For MR2, you just have to compile Giraph with the new hadoop_2 profile and then
> it should work with no problems using the commands shown in the Giraph
> quickstart.
>
> For Yarn, compile with the hadoop_yarn profile and the application submission
> command follows the following template (replace variables with your particular
> values):
>
> GIRAPH_JAR_FOLDER=/home/alex/Giraph
> GIRAPH_JAR_NAME=giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-
> dependencies.jar
> GIRAPH_JAR=$GIRAPH_JAR_FOLDER/$GIRAPH_JAR_NAME
>
> HADOOP_CLASSPATH=$GIRAPH_JAR_FOLDER hadoop jar $GIRAPH_JAR
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
> /user/alex/input/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/alex/output/shortestpaths -w 1 -yj $GIRAPH_JAR_NAME
>
> If you don't want to compile things yourself, the grafos.ml in which I'm
> working at the moment provides, among other things, pre-compiled versions of
> giraph straight from trunk for the major targets integrated with some machine
> learning algorithms. You can check the download page here:
> http://grafos.ml/Download.html
>
> We also have a configuration tool that should make it easy to find the exact
> command to run different algorithms (including most of Giraph examples):
> http://grafos.ml/configure.html
>
> Hopefully this should be sufficient for you to successfully run Giraph in 2.2.0.
>
> Best regards,
> Alex
>
> On Sunday, March 09, 2014 08:59:32 PM Vishal Mishra wrote:
>> Hi Guys,
>> I tried giraph on hadoop-2.2.0 cluster a few months back without success
>> and moved onto other pressing issues but would really love to try out again.
>>
>> Has anyone been able to run giraph successfully on a hadoop-2.2.0 cluster?
>> if yes, I'd really appreciate it if you can share the steps to run the
>> "SimplePageRankComputation" example.
>> If not, please share the steps, I'd give it a try and provide any required
>> logs for debugging.
>> Our setup (works for our regular hadoop/hive jobs):
>> - ubuntu 13.10
>> - 1 namenode & resourcemanager. + zookeeper
>> - 3 datanode & nodemanager
>>
>> Thanx,
>> Vishal
Re: Steps for Giraph on hadoop-2.2.0 Cluster
Posted by Alexandre Fonseca <al...@gmail.com>.
Hello Vishal,
I've run it successfully in hadoop-2.2.0 with both MR2 and Yarn computations.
For MR2, you just have to compile Giraph with the new hadoop_2 profile and then
it should work with no problems using the commands shown in the Giraph
quickstart.
For Yarn, compile with the hadoop_yarn profile and the application submission
command follows the following template (replace variables with your particular
values):
GIRAPH_JAR_FOLDER=/home/alex/Giraph
GIRAPH_JAR_NAME=giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-
dependencies.jar
GIRAPH_JAR=$GIRAPH_JAR_FOLDER/$GIRAPH_JAR_NAME
HADOOP_CLASSPATH=$GIRAPH_JAR_FOLDER hadoop jar $GIRAPH_JAR
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
/user/alex/input/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/alex/output/shortestpaths -w 1 -yj $GIRAPH_JAR_NAME
If you don't want to compile things yourself, the grafos.ml in which I'm
working at the moment provides, among other things, pre-compiled versions of
giraph straight from trunk for the major targets integrated with some machine
learning algorithms. You can check the download page here:
http://grafos.ml/Download.html
We also have a configuration tool that should make it easy to find the exact
command to run different algorithms (including most of Giraph examples):
http://grafos.ml/configure.html
Hopefully this should be sufficient for you to successfully run Giraph in 2.2.0.
Best regards,
Alex
On Sunday, March 09, 2014 08:59:32 PM Vishal Mishra wrote:
> Hi Guys,
> I tried giraph on hadoop-2.2.0 cluster a few months back without success
> and moved onto other pressing issues but would really love to try out again.
>
> Has anyone been able to run giraph successfully on a hadoop-2.2.0 cluster?
> if yes, I'd really appreciate it if you can share the steps to run the
> "SimplePageRankComputation" example.
> If not, please share the steps, I'd give it a try and provide any required
> logs for debugging.
> Our setup (works for our regular hadoop/hive jobs):
> - ubuntu 13.10
> - 1 namenode & resourcemanager. + zookeeper
> - 3 datanode & nodemanager
>
> Thanx,
> Vishal