You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Eli Reisman <ap...@gmail.com> on 2013/12/08 19:23:31 UTC
Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)
Hi all,
It looks to me like you want to run the giraph-core jar at the beginning of
the "hadoop jar" call and supply the giraph-examples jar with -yj argument.
Also, you can place any -D type configuration settings you're sick of
supplying at the command line into giraph-site.xml just as you would with
Hadoop settings in one of its own site.xml files, and place these where
your local Hadoop client configs are and they should be picked up in your
application.
Hope that helps,
Eli
On Thu, Nov 28, 2013 at 12:21 PM, Vishal Mishra <mi...@gmail.com>wrote:
> I had a mistake in my command (moronic copy/paste zookeeper settings
> error).
> Updated and now hitting the following:
>
> ....
> 13/11/28 12:02:14 INFO yarn.GiraphYarnClient: Running Client
> 13/11/28 12:02:14 INFO client.RMProxy: Connecting to ResourceManager at
> master/192.168.1.100:8045
> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
> for, nodeId=kreator:46477, nodeAddresskreator:8042,
> nodeRackName/default-rack, nodeNumContainers7
> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
> for, nodeId=exotica:46645, nodeAddressexotica:8042,
> nodeRackName/default-rack, nodeNumContainers8
> Exception in thread "main" java.lang.RuntimeException: Giraph job requires
> 2 containers to run; cluster only hosts 15
> at
> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230)
> at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:125)
> ....
>
> It seems like some config is missing from our hadoop setup.
> Also, note that I'm working from the trunk branch.
>
> Thanx,
> Vishal
>
>
> On Wed, Nov 27, 2013 at 11:34 AM, Vishal Mishra <mi...@gmail.com>wrote:
>
>> Hey Guys,
>> I'm trying giraph 1.0.0 on hadoop 2.2.0 (2-node cluster) but hitting the
>> same problem as outlined in this thread. Was there a resolution? If yes,
>> can you please share?
>> Completely new to giraph and hadoop so please excuse any omissions. Our
>> cluster is fully functional and running mapreduce and hive jobs cleanly.
>>
>> Command:
>> hadoop jar
>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner -D giraph.zkList="zkNode.net:2081"
>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> -vip /user/hduser/rrdata/tiny_graph.txt -vof
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> /user/hduser/rrdata/output -w 1 -yj
>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>
>> Trimmed Console Output:
>> 13/11/27 11:19:58 INFO yarn.GiraphYarnClient:
>> appattempt_1385518875931_0002_000001, State: ACCEPTED, Containers used: 1
>> 13/11/27 11:20:00 ERROR yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>> state, diagnostics show: Application application_1385518875931_0002 failed
>> 2 times due to AM Container for appattempt_1385518875931_0002_000002 exited
>> with exitCode: 1 due to: Exception from container-launch:
>> org.apache.hadoop.util.Shell$ExitCodeException:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>>
>>
>> Thanx,
>> Vishal
>>
>>
>> On Tue, Nov 19, 2013 at 3:16 PM, Mohammad Islam <mi...@yahoo.com>wrote:
>>
>>> Rob,
>>> I understand your frustration.
>>> I'm pre-occupied with some other stuffs in next couple of days.
>>> If no one replied by then, I will recreate the situation here and try to
>>> give you more concrete help.
>>>
>>> Regards,
>>> Mohammad
>>>
>>>
>>> On Tuesday, November 19, 2013 2:56 AM, Rob Vesse <rv...@dotnetrdf.org>
>>> wrote:
>>> Mohammad
>>>
>>> Comments inline:
>>>
>>> From: Mohammad Islam <mi...@yahoo.com>
>>> Reply-To: <us...@giraph.apache.org>
>>> Date: Monday, 18 November 2013 22:24
>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop
>>> 2.2.0)
>>>
>>> I used the following shell script to run PR where my current working
>>> directory contains the graph*.jar in Hadoop 2.1.1.
>>> . You can try something similar.
>>>
>>>
>>> I already am.
>>>
>>>
>>>
>>>
>>> HAD_VERSION=2.1.1-SNAPSHOT
>>> hadoop fs -rmr out
>>> export GIRAPH_HOME=/home/mislam/softwares/giraph
>>> #export
>>> GIRAPH_CORE=$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>> #export
>>> GIRAPH_EXAMPLE=$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>
>>>
>>> CLASSPATH=$CLASSPATH:$GIRAPH_HOME/conf
>>>
>>>
>>> What is the point of putting the conf directory on the CLASSPATH? I
>>> assume this is only to pick up the giraph configuration but at the same
>>> time there is nothing to stop me putting my Giraph specific config in one
>>> of the other Hadoop configuration files e.g. core-site.xml?
>>>
>>> #CLASSPATH=$CLASSPATH:$GIRAPH_CORE
>>> #CLASSPATH=$CLASSPATH:$GIRAPH_EXAMPLE
>>>
>>> export CLASSPATH=$CLASSPATH
>>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
>>>
>>> echo $HADOOP_CLASSPATH
>>>
>>>
>>> I assume the point of the above is to ensure that the Giraph config file
>>> is on the class path Hadoop actually uses but adding this makes no
>>> difference to me, I get exactly the same failure as before. Modifying the
>>> logic to add the examples JAR onto the class path does not help either.
>>>
>>>
>>>
>>> hadoop --config $HADOOP_HOME/etc/hadoop jar
>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>> org.apache.giraph.GiraphRunner
>>> org.apache.giraph.examples.SimplePageRankComputation -c
>>> org.apache.giraph.combiner.DoubleSumCombiner -w 1 -yh 1024 -yj
>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>> -vif
>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -mc
>>> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
>>> -vip input -op out
>>>
>>>
>>> This is not the same command that I am running, also DoubleSumCombiner
>>> gives a ClassNotFoundException, you actually want DoubleSumMessageCombiner.
>>> If I adapt my script to run your command I experience the exact same error
>>> as I do trying to run the command I showed so I am still nowhere.
>>>
>>> I'm getting a little frustrated that no-one has any idea what the
>>> problem is here, either this is a simple misconfiguration on my part or a
>>> bug in Giraph for Hadoop 2.2.0 ? Given the thread so far and the fact that
>>> I can run MR based Giraph on Hadoop 1.2.1 this is looking increasingly
>>> unlikely I'm starting to lean heavily towards the latter option.
>>>
>>> Rob
>>>
>>>
>>>
>>> On Monday, November 18, 2013 2:24 AM, Rob Vesse <rv...@dotnetrdf.org>
>>> wrote:
>>> Hey all
>>>
>>> Bumping this thread because I am still completely blocked on this and
>>> hoping someone can help me. As explained earlier in the thread I have
>>> Giraph built for pure YARN on stable Hadoop 2.2.0 release like so:
>>>
>>> mvn –Phadoop_yarn –Dhadoop.version=2.2.0 –DskipTests=true clean install
>>>
>>> But when I try and run the simple shortest paths demo like so:
>>>
>>> $> bin/hadoop jar
>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>> org.apache.giraph.GiraphRunner
>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>> -vip /user/rvesse/tiny_graph.txt -vof
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>> /user/rvesse/shortestpaths -w 1 -yj
>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>
>>> The application fails and the YARN logs show the following error:
>>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>
>>>
>>>
>>> Any ideas why the GiraphApplicationMaster would not be found? Have I just got something basic wrong in my configuration or is this a bug in Giraph?
>>>
>>>
>>> I found an earlier thread on the same topic (http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c51E5A4D1.40006@googlemail.com%3e) but it was never resolved hence why I have now changed the topic in the hope it will jog someone's memory
>>>
>>>
>>> Cheers,
>>>
>>>
>>> Rob
>>>
>>>
>>> From: Rob Vesse <rv...@dotnetrdf.org>
>>> Reply-To: <us...@giraph.apache.org>
>>> Date: Friday, 15 November 2013 14:14
>>> To: <us...@giraph.apache.org>
>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>
>>> Adding the –yj argument to specify the YARN JARs doesn't make any
>>> difference.
>>>
>>> Here is the complete console output from my run:
>>>
>>> bin/hadoop jar
>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>> org.apache.giraph.GiraphRunner
>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>> -vip /user/rvesse/tiny_graph.txt -vof
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>> /user/rvesse/shortestpaths/3 -w 1 -yj
>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>> 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info from
>>> SCDynamicStore
>>> 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load
>>> native-hadoop library for your platform... using builtin-java classes where
>>> applicable
>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format
>>> specified. Ensure your InputFormat does not require one.
>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output format
>>> specified. Ensure your OutputFormat does not require one.
>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is:
>>> hdfs://localhost/user/rvesse/shortestpaths/3
>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
>>> 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager at /
>>> 0.0.0.0:8032
>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from ASM
>>> for, nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042,
>>> nodeRackName/default-rack, nodeNumContainers0
>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task heap
>>> size to 1024MB.
>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application
>>> ID: application_1384523910015_0004
>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is
>>> deprecated. Instead, use mapreduce.job.id
>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for
>>> the application master
>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM
>>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir is
>>> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in LocalResources
>>> ::
>>> hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_1384523910015_0004/giraph-conf.xml
>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS
>>> :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>> /contrib/capacity-scheduler/*.jar
>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>> /contrib/capacity-scheduler
>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient:
>>> ApplicationSumbissionContext for GiraphApplicationMaster launch container
>>> is populated.
>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application to
>>> ASM
>>> 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application
>>> application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after
>>> submission :application_1384523910015_0004
>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>>> container request was submitted to ResourceManager for job: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation
>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.82
>>> secs
>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
>>> appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers used: 1
>>> 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>> state, diagnostics show: Application application_1384523910015_0004 failed
>>> 2 times due to AM Container for appattempt_1384523910015_0004_000002 exited
>>> with exitCode: 1 due to: Exception from container-launch:
>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>> at
>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>> at java.lang.Thread.run(Thread.java:695)
>>>
>>>
>>> .Failing this attempt.. Failing the application.
>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS
>>> distributed cache directory for Giraph job.
>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
>>> running time: 0 minutes, 3 seconds.
>>>
>>> And I see the same error in the logs:
>>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>
>>>
>>> Any more ideas?
>>>
>>>
>>>
>>> Rob
>>>
>>>
>>> From: Mohammad Islam <mi...@yahoo.com>
>>> Reply-To: <us...@giraph.apache.org>
>>> Date: Thursday, 14 November 2013 20:36
>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>
>>> I tried the following command few months back. I found you are not
>>> passing some params.
>>>
>>> bin/hadoop --config etc/hadoop jar
>>> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-examples-1.?.0-for-hadoop-???-jar-with-dependencies.jar
>>> org.apache.giraph.GiraphRunner
>>> org.apache.giraph.examples.SimplePageRankVertexOpt -c
>>> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj
>>> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif
>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -of
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip
>>> /user/me/input -op /user/me/output_35w_20G_80m_1
>>>
>>>
>>> On Thursday, November 14, 2013 3:22 AM, Rob Vesse <
>>> rvesse@dotnetrdf.org> wrote:
>>> It's the same command line I showed earlier, just trying to run the
>>> example application:
>>>
>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>> org.apache.giraph.GiraphRunner
>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>> -vip /user/rvesse/tiny_graph.txt -vof
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>> /user/rvesse/shortestpaths/2 -w 1
>>>
>>> Rob
>>>
>>> From: Mohammad Islam <mi...@yahoo.com>
>>> Reply-To: <us...@giraph.apache.org>
>>> Date: Thursday, 14 November 2013 05:57
>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>
>>> What are you running? PR?
>>> What is the command line?
>>>
>>>
>>>
>>> On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <
>>> rvesse@dotnetrdf.org> wrote:
>>> So I fixed the previous issue I was having which I reported and
>>> submitted a patch as GIRAPH-795
>>>
>>> I then went ahead with installing and configuring my external Zookeeper
>>> appropriately and tried running again but now the YARN application fails
>>> and I find the following error in the logs:
>>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> org/apache/giraph/yarn/GiraphApplicationMaster
>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>
>>>
>>> Anybody have idea what the cause of this one might be or how to fix it?
>>>
>>>
>>> Cheers,
>>>
>>>
>>> Rob
>>>
>>>
>>> From: Rob Vesse <rv...@dotnetrdf.org>
>>> Reply-To: <us...@giraph.apache.org>
>>> Date: Wednesday, 13 November 2013 15:35
>>> To: <us...@giraph.apache.org>
>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>
>>> Mohammad
>>>
>>> This got me a build that at least attempts to start to run but not I see
>>> the following:
>>>
>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>> org.apache.giraph.GiraphRunner
>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>> -vip /user/rvesse/tiny_graph.txt -vof
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>> /user/rvesse/shortestpaths/2 -w 1
>>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info from
>>> SCDynamicStore
>>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load
>>> native-hadoop library for your platform... using builtin-java classes where
>>> applicable
>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format
>>> specified. Ensure your InputFormat does not require one.
>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output format
>>> specified. Ensure your OutputFormat does not require one.
>>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is:
>>> hdfs://localhost/user/rvesse/shortestpaths/2
>>> Exception in thread "main" java.lang.NullPointerException
>>> at
>>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(GiraphYarnClient.java:460)
>>> at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>
>>> Which indirectly implies that I have some missing configuration to run
>>> Giraph on YARN. I don't think that should be a NPE and it looks to be a
>>> trivial fix so I'll try and fix and send a pull request for that.
>>>
>>> The error message I should receive if the NPE wasn't happening is the
>>> following:
>>>
>>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances:
>>> use a standalone ZooKeeper.
>>>
>>> What configuration options do I need to have set to point Giraph at an
>>> external Zookeeper?
>>>
>>> Cheers,
>>>
>>> Rob
>>>
>>> From: Mohammad Islam <mi...@yahoo.com>
>>> Reply-To: <us...@giraph.apache.org>
>>> Date: Wednesday, 13 November 2013 11:15
>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>
>>> Hi Rob,
>>> Did you try with this command "mvn -Phadoop_yarn -Dhadoop.version=2.2.0
>>> clean package -DskipTests"?
>>> Regards,
>>> Mohammad
>>>
>>>
>>> On Wednesday, November 13, 2013 2:55 AM, Rob Vesse <
>>> rvesse@dotnetrdf.org> wrote:
>>> Hi all
>>>
>>> I'm having trouble getting Giraph to build and run against a Hadoop
>>> 2.2.0 installation. I tried the hadoop_2.0.3 profile with no luck because
>>> I get class cast errors when trying to run the examples.
>>>
>>> Is there a specific profile I should be building to support Hadoop 2.2.0
>>> or is this not supported yet?
>>>
>>> Thanks,
>>>
>>> Rob
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)
Posted by Rafal Wojdyla <ra...@gmail.com>.
@Vishal - please try: https://issues.apache.org/jira/browse/GIRAPH-819
On Mon, Dec 9, 2013 at 9:56 PM, Vishal Mishra <mi...@gmail.com>wrote:
> Rafal, thanx for pointing that out earlier and it was the problem since we
> were running a fairly intensive job but re-trying got me back to the
> previous error.
>
> Eli,
> I tried your suggestion but it leads to java.lang.ClassNotFoundException
> for SimpleShortestPathsComputation as it's not defined there. Maybe I
> didn't understand your suggestion?
> Thanx for the pointer to add the external zookeeper setting to
> giraph-site.xml esp. since following Alexandre Fonseca's suggestion leads
> to different command line options.
>
> Alex, I've replied to the other thread where I'm hitting the same error as
> Rohit.
>
>
>
> On Mon, Dec 9, 2013 at 3:42 AM, Rob Vesse <rv...@dotnetrdf.org> wrote:
>
>> Yes it actually is in the JAR file:
>>
>> mbp-rvesse:target rvesse$ jar -tf
>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> | grep org.apache.giraph.yarn.GiraphApplicationMaster
>> org/apache/giraph/yarn/GiraphApplicationMaster$1.class
>>
>> org/apache/giraph/yarn/GiraphApplicationMaster$LaunchContainerRunnable.class
>> org/apache/giraph/yarn/GiraphApplicationMaster$NMCallbackHandler.class
>> org/apache/giraph/yarn/GiraphApplicationMaster$RMCallbackHandler.class
>> org/apache/giraph/yarn/GiraphApplicationMaster.class
>>
>> Rob
>>
>> From: Rafal Wojdyla <ra...@gmail.com>
>> Reply-To: <us...@giraph.apache.org>
>> Date: Monday, 9 December 2013 10:24
>> To: <us...@giraph.apache.org>
>>
>> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop
>> 2.2.0)
>>
>> Hi all!
>>
>> I had the same problems as you guys:
>>
>> @Rob:
>> Did you check if *org.apache.giraph.yarn.GiraphApplicationMaster* is actually
>> in the jar file - it wasn't it my case - I'm using custom profile in maven.
>> Do: *jar -tf
>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> | grep org.apache.giraph.yarn.GiraphApplicationMaster*
>>
>> @Vishal:
>> I got this error few times when cluster was highly utilized, problem
>> disappeared when I used queue with free resources - this may be a problem
>> but I haven't investigated it.
>>
>> Unfortunately I wasn't able to successfully run Giraph application
>> anyway, due to: https://issues.apache.org/jira/browse/GIRAPH-806
>>
>> Cheers - Rafal
>>
>>
>> On Sun, Dec 8, 2013 at 7:23 PM, Eli Reisman <ap...@gmail.com>wrote:
>>
>>> Hi all,
>>>
>>> It looks to me like you want to run the giraph-core jar at the beginning
>>> of the "hadoop jar" call and supply the giraph-examples jar with -yj
>>> argument.
>>>
>>> Also, you can place any -D type configuration settings you're sick of
>>> supplying at the command line into giraph-site.xml just as you would with
>>> Hadoop settings in one of its own site.xml files, and place these where
>>> your local Hadoop client configs are and they should be picked up in your
>>> application.
>>>
>>> Hope that helps,
>>>
>>> Eli
>>>
>>>
>>>
>>> On Thu, Nov 28, 2013 at 12:21 PM, Vishal Mishra <mi...@gmail.com>wrote:
>>>
>>>> I had a mistake in my command (moronic copy/paste zookeeper settings
>>>> error).
>>>> Updated and now hitting the following:
>>>>
>>>> ....
>>>> 13/11/28 12:02:14 INFO yarn.GiraphYarnClient: Running Client
>>>> 13/11/28 12:02:14 INFO client.RMProxy: Connecting to ResourceManager at
>>>> master/192.168.1.100:8045
>>>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
>>>> for, nodeId=kreator:46477, nodeAddresskreator:8042,
>>>> nodeRackName/default-rack, nodeNumContainers7
>>>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
>>>> for, nodeId=exotica:46645, nodeAddressexotica:8042,
>>>> nodeRackName/default-rack, nodeNumContainers8
>>>> Exception in thread "main" java.lang.RuntimeException: Giraph job
>>>> requires 2 containers to run; cluster only hosts 15
>>>> at
>>>> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230)
>>>> at
>>>> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:125)
>>>> ....
>>>>
>>>> It seems like some config is missing from our hadoop setup.
>>>> Also, note that I'm working from the trunk branch.
>>>>
>>>> Thanx,
>>>> Vishal
>>>>
>>>>
>>>> On Wed, Nov 27, 2013 at 11:34 AM, Vishal Mishra <mishravishal@gmail.com
>>>> > wrote:
>>>>
>>>>> Hey Guys,
>>>>> I'm trying giraph 1.0.0 on hadoop 2.2.0 (2-node cluster) but hitting
>>>>> the same problem as outlined in this thread. Was there a resolution? If
>>>>> yes, can you please share?
>>>>> Completely new to giraph and hadoop so please excuse any omissions.
>>>>> Our cluster is fully functional and running mapreduce and hive jobs cleanly.
>>>>>
>>>>> Command:
>>>>> hadoop jar
>>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>> org.apache.giraph.GiraphRunner -D giraph.zkList="zkNode.net:2081"
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>> -vip /user/hduser/rrdata/tiny_graph.txt -vof
>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>> /user/hduser/rrdata/output -w 1 -yj
>>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>
>>>>> Trimmed Console Output:
>>>>> 13/11/27 11:19:58 INFO yarn.GiraphYarnClient:
>>>>> appattempt_1385518875931_0002_000001, State: ACCEPTED, Containers used: 1
>>>>> 13/11/27 11:20:00 ERROR yarn.GiraphYarnClient: Giraph:
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>>>> state, diagnostics show: Application application_1385518875931_0002 failed
>>>>> 2 times due to AM Container for appattempt_1385518875931_0002_000002 exited
>>>>> with exitCode: 1 due to: Exception from container-launch:
>>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>>> at
>>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>
>>>>>
>>>>> Thanx,
>>>>> Vishal
>>>>>
>>>>>
>>>>> On Tue, Nov 19, 2013 at 3:16 PM, Mohammad Islam <mi...@yahoo.com>wrote:
>>>>>
>>>>>> Rob,
>>>>>> I understand your frustration.
>>>>>> I'm pre-occupied with some other stuffs in next couple of days.
>>>>>> If no one replied by then, I will recreate the situation here and try
>>>>>> to give you more concrete help.
>>>>>>
>>>>>> Regards,
>>>>>> Mohammad
>>>>>>
>>>>>>
>>>>>> On Tuesday, November 19, 2013 2:56 AM, Rob Vesse <
>>>>>> rvesse@dotnetrdf.org> wrote:
>>>>>> Mohammad
>>>>>>
>>>>>> Comments inline:
>>>>>>
>>>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>>>> Reply-To: <us...@giraph.apache.org>
>>>>>> Date: Monday, 18 November 2013 22:24
>>>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>>>> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and
>>>>>> Hadoop 2.2.0)
>>>>>>
>>>>>> I used the following shell script to run PR where my current working
>>>>>> directory contains the graph*.jar in Hadoop 2.1.1.
>>>>>> . You can try something similar.
>>>>>>
>>>>>>
>>>>>> I already am.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> HAD_VERSION=2.1.1-SNAPSHOT
>>>>>> hadoop fs -rmr out
>>>>>> export GIRAPH_HOME=/home/mislam/softwares/giraph
>>>>>> #export
>>>>>> GIRAPH_CORE=$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>> #export
>>>>>> GIRAPH_EXAMPLE=$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>>
>>>>>>
>>>>>> CLASSPATH=$CLASSPATH:$GIRAPH_HOME/conf
>>>>>>
>>>>>>
>>>>>> What is the point of putting the conf directory on the CLASSPATH? I
>>>>>> assume this is only to pick up the giraph configuration but at the same
>>>>>> time there is nothing to stop me putting my Giraph specific config in one
>>>>>> of the other Hadoop configuration files e.g. core-site.xml?
>>>>>>
>>>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_CORE
>>>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_EXAMPLE
>>>>>>
>>>>>> export CLASSPATH=$CLASSPATH
>>>>>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
>>>>>>
>>>>>> echo $HADOOP_CLASSPATH
>>>>>>
>>>>>>
>>>>>> I assume the point of the above is to ensure that the Giraph config
>>>>>> file is on the class path Hadoop actually uses but adding this makes no
>>>>>> difference to me, I get exactly the same failure as before. Modifying the
>>>>>> logic to add the examples JAR onto the class path does not help either.
>>>>>>
>>>>>>
>>>>>>
>>>>>> hadoop --config $HADOOP_HOME/etc/hadoop jar
>>>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimplePageRankComputation -c
>>>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 1 -yh 1024 -yj
>>>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>> -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -mc
>>>>>> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
>>>>>> -vip input -op out
>>>>>>
>>>>>>
>>>>>> This is not the same command that I am running, also
>>>>>> DoubleSumCombiner gives a ClassNotFoundException, you actually want
>>>>>> DoubleSumMessageCombiner. If I adapt my script to run your command I
>>>>>> experience the exact same error as I do trying to run the command I showed
>>>>>> so I am still nowhere.
>>>>>>
>>>>>> I'm getting a little frustrated that no-one has any idea what the
>>>>>> problem is here, either this is a simple misconfiguration on my part or a
>>>>>> bug in Giraph for Hadoop 2.2.0 ? Given the thread so far and the fact that
>>>>>> I can run MR based Giraph on Hadoop 1.2.1 this is looking increasingly
>>>>>> unlikely I'm starting to lean heavily towards the latter option.
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Monday, November 18, 2013 2:24 AM, Rob Vesse <
>>>>>> rvesse@dotnetrdf.org> wrote:
>>>>>> Hey all
>>>>>>
>>>>>> Bumping this thread because I am still completely blocked on this and
>>>>>> hoping someone can help me. As explained earlier in the thread I have
>>>>>> Giraph built for pure YARN on stable Hadoop 2.2.0 release like so:
>>>>>>
>>>>>> mvn –Phadoop_yarn –Dhadoop.version=2.2.0 –DskipTests=true clean
>>>>>> install
>>>>>>
>>>>>> But when I try and run the simple shortest paths demo like so:
>>>>>>
>>>>>> $> bin/hadoop jar
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>> /user/rvesse/shortestpaths -w 1 -yj
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>
>>>>>> The application fails and the YARN logs show the following error:
>>>>>>
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>
>>>>>>
>>>>>> Any ideas why the GiraphApplicationMaster would not be found? Have I just got something basic wrong in my configuration or is this a bug in Giraph?
>>>>>>
>>>>>>
>>>>>> I found an earlier thread on the same topic (http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c51E5A4D1.40006@googlemail.com%3e) but it was never resolved hence why I have now changed the topic in the hope it will jog someone's memory
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>>
>>>>>> From: Rob Vesse <rv...@dotnetrdf.org>
>>>>>> Reply-To: <us...@giraph.apache.org>
>>>>>> Date: Friday, 15 November 2013 14:14
>>>>>> To: <us...@giraph.apache.org>
>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>
>>>>>> Adding the –yj argument to specify the YARN JARs doesn't make any
>>>>>> difference.
>>>>>>
>>>>>> Here is the complete console output from my run:
>>>>>>
>>>>>> bin/hadoop jar
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>> /user/rvesse/shortestpaths/3 -w 1 -yj
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info
>>>>>> from SCDynamicStore
>>>>>> 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load
>>>>>> native-hadoop library for your platform... using builtin-java classes where
>>>>>> applicable
>>>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format
>>>>>> specified. Ensure your InputFormat does not require one.
>>>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output
>>>>>> format specified. Ensure your OutputFormat does not require one.
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is:
>>>>>> hdfs://localhost/user/rvesse/shortestpaths/3
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
>>>>>> 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager
>>>>>> at /0.0.0.0:8032
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from
>>>>>> ASM for, nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042,
>>>>>> nodeRackName/default-rack, nodeNumContainers0
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task
>>>>>> heap size to 1024MB.
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new
>>>>>> Application ID: application_1384523910015_0004
>>>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is
>>>>>> deprecated. Instead, use mapreduce.job.id
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for
>>>>>> the application master
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM
>>>>>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap
>>>>>> ....
>>>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir
>>>>>> is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in
>>>>>> LocalResources ::
>>>>>> hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_1384523910015_0004/giraph-conf.xml
>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS
>>>>>> :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>> /contrib/capacity-scheduler/*.jar
>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>> /contrib/capacity-scheduler
>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient:
>>>>>> ApplicationSumbissionContext for GiraphApplicationMaster launch container
>>>>>> is populated.
>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application
>>>>>> to ASM
>>>>>> 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application
>>>>>> application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after
>>>>>> submission :application_1384523910015_0004
>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>>>>>> container request was submitted to ResourceManager for job: Giraph:
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation
>>>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.82
>>>>>> secs
>>>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
>>>>>> appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers used: 1
>>>>>> 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>>>>> state, diagnostics show: Application application_1384523910015_0004 failed
>>>>>> 2 times due to AM Container for appattempt_1384523910015_0004_000002 exited
>>>>>> with exitCode: 1 due to: Exception from container-launch:
>>>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>>>> at
>>>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>>>>> at
>>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>>>> at
>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>>>>> at
>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>>>>> at
>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>>> at java.lang.Thread.run(Thread.java:695)
>>>>>>
>>>>>>
>>>>>> .Failing this attempt.. Failing the application.
>>>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS
>>>>>> distributed cache directory for Giraph job.
>>>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
>>>>>> running time: 0 minutes, 3 seconds.
>>>>>>
>>>>>> And I see the same error in the logs:
>>>>>>
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>
>>>>>>
>>>>>> Any more ideas?
>>>>>>
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>>
>>>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>>>> Reply-To: <us...@giraph.apache.org>
>>>>>> Date: Thursday, 14 November 2013 20:36
>>>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>
>>>>>> I tried the following command few months back. I found you are not
>>>>>> passing some params.
>>>>>>
>>>>>> bin/hadoop --config etc/hadoop jar
>>>>>> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-examples-1.?.0-for-hadoop-???-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimplePageRankVertexOpt -c
>>>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj
>>>>>> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -of
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip
>>>>>> /user/me/input -op /user/me/output_35w_20G_80m_1
>>>>>>
>>>>>>
>>>>>> On Thursday, November 14, 2013 3:22 AM, Rob Vesse <
>>>>>> rvesse@dotnetrdf.org> wrote:
>>>>>> It's the same command line I showed earlier, just trying to run the
>>>>>> example application:
>>>>>>
>>>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>> /user/rvesse/shortestpaths/2 -w 1
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>>>> Reply-To: <us...@giraph.apache.org>
>>>>>> Date: Thursday, 14 November 2013 05:57
>>>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>
>>>>>> What are you running? PR?
>>>>>> What is the command line?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <
>>>>>> rvesse@dotnetrdf.org> wrote:
>>>>>> So I fixed the previous issue I was having which I reported and
>>>>>> submitted a patch as GIRAPH-795
>>>>>>
>>>>>> I then went ahead with installing and configuring my external
>>>>>> Zookeeper appropriately and tried running again but now the YARN
>>>>>> application fails and I find the following error in the logs:
>>>>>>
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>> org/apache/giraph/yarn/GiraphApplicationMaster
>>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>
>>>>>>
>>>>>> Anybody have idea what the cause of this one might be or how to fix it?
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>>
>>>>>> From: Rob Vesse <rv...@dotnetrdf.org>
>>>>>> Reply-To: <us...@giraph.apache.org>
>>>>>> Date: Wednesday, 13 November 2013 15:35
>>>>>> To: <us...@giraph.apache.org>
>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>
>>>>>> Mohammad
>>>>>>
>>>>>> This got me a build that at least attempts to start to run but not I
>>>>>> see the following:
>>>>>>
>>>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>> /user/rvesse/shortestpaths/2 -w 1
>>>>>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info
>>>>>> from SCDynamicStore
>>>>>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load
>>>>>> native-hadoop library for your platform... using builtin-java classes where
>>>>>> applicable
>>>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format
>>>>>> specified. Ensure your InputFormat does not require one.
>>>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output
>>>>>> format specified. Ensure your OutputFormat does not require one.
>>>>>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is:
>>>>>> hdfs://localhost/user/rvesse/shortestpaths/2
>>>>>> Exception in thread "main" java.lang.NullPointerException
>>>>>> at
>>>>>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(GiraphYarnClient.java:460)
>>>>>> at
>>>>>> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>>>
>>>>>> Which indirectly implies that I have some missing configuration to
>>>>>> run Giraph on YARN. I don't think that should be a NPE and it looks to be
>>>>>> a trivial fix so I'll try and fix and send a pull request for that.
>>>>>>
>>>>>> The error message I should receive if the NPE wasn't happening is the
>>>>>> following:
>>>>>>
>>>>>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances:
>>>>>> use a standalone ZooKeeper.
>>>>>>
>>>>>> What configuration options do I need to have set to point Giraph at
>>>>>> an external Zookeeper?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>>>> Reply-To: <us...@giraph.apache.org>
>>>>>> Date: Wednesday, 13 November 2013 11:15
>>>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>
>>>>>> Hi Rob,
>>>>>> Did you try with this command "mvn -Phadoop_yarn
>>>>>> -Dhadoop.version=2.2.0 clean package -DskipTests"?
>>>>>> Regards,
>>>>>> Mohammad
>>>>>>
>>>>>>
>>>>>> On Wednesday, November 13, 2013 2:55 AM, Rob Vesse <
>>>>>> rvesse@dotnetrdf.org> wrote:
>>>>>> Hi all
>>>>>>
>>>>>> I'm having trouble getting Giraph to build and run against a Hadoop
>>>>>> 2.2.0 installation. I tried the hadoop_2.0.3 profile with no luck because
>>>>>> I get class cast errors when trying to run the examples.
>>>>>>
>>>>>> Is there a specific profile I should be building to support Hadoop
>>>>>> 2.2.0 or is this not supported yet?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)
Posted by Vishal Mishra <mi...@gmail.com>.
Rafal, thanx for pointing that out earlier and it was the problem since we
were running a fairly intensive job but re-trying got me back to the
previous error.
Eli,
I tried your suggestion but it leads to java.lang.ClassNotFoundException
for SimpleShortestPathsComputation as it's not defined there. Maybe I
didn't understand your suggestion?
Thanx for the pointer to add the external zookeeper setting to
giraph-site.xml esp. since following Alexandre Fonseca's suggestion leads
to different command line options.
Alex, I've replied to the other thread where I'm hitting the same error as
Rohit.
On Mon, Dec 9, 2013 at 3:42 AM, Rob Vesse <rv...@dotnetrdf.org> wrote:
> Yes it actually is in the JAR file:
>
> mbp-rvesse:target rvesse$ jar -tf
> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> | grep org.apache.giraph.yarn.GiraphApplicationMaster
> org/apache/giraph/yarn/GiraphApplicationMaster$1.class
>
> org/apache/giraph/yarn/GiraphApplicationMaster$LaunchContainerRunnable.class
> org/apache/giraph/yarn/GiraphApplicationMaster$NMCallbackHandler.class
> org/apache/giraph/yarn/GiraphApplicationMaster$RMCallbackHandler.class
> org/apache/giraph/yarn/GiraphApplicationMaster.class
>
> Rob
>
> From: Rafal Wojdyla <ra...@gmail.com>
> Reply-To: <us...@giraph.apache.org>
> Date: Monday, 9 December 2013 10:24
> To: <us...@giraph.apache.org>
>
> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop
> 2.2.0)
>
> Hi all!
>
> I had the same problems as you guys:
>
> @Rob:
> Did you check if *org.apache.giraph.yarn.GiraphApplicationMaster* is actually
> in the jar file - it wasn't it my case - I'm using custom profile in maven.
> Do: *jar -tf
> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> | grep org.apache.giraph.yarn.GiraphApplicationMaster*
>
> @Vishal:
> I got this error few times when cluster was highly utilized, problem
> disappeared when I used queue with free resources - this may be a problem
> but I haven't investigated it.
>
> Unfortunately I wasn't able to successfully run Giraph application anyway,
> due to: https://issues.apache.org/jira/browse/GIRAPH-806
>
> Cheers - Rafal
>
>
> On Sun, Dec 8, 2013 at 7:23 PM, Eli Reisman <ap...@gmail.com>wrote:
>
>> Hi all,
>>
>> It looks to me like you want to run the giraph-core jar at the beginning
>> of the "hadoop jar" call and supply the giraph-examples jar with -yj
>> argument.
>>
>> Also, you can place any -D type configuration settings you're sick of
>> supplying at the command line into giraph-site.xml just as you would with
>> Hadoop settings in one of its own site.xml files, and place these where
>> your local Hadoop client configs are and they should be picked up in your
>> application.
>>
>> Hope that helps,
>>
>> Eli
>>
>>
>>
>> On Thu, Nov 28, 2013 at 12:21 PM, Vishal Mishra <mi...@gmail.com>wrote:
>>
>>> I had a mistake in my command (moronic copy/paste zookeeper settings
>>> error).
>>> Updated and now hitting the following:
>>>
>>> ....
>>> 13/11/28 12:02:14 INFO yarn.GiraphYarnClient: Running Client
>>> 13/11/28 12:02:14 INFO client.RMProxy: Connecting to ResourceManager at
>>> master/192.168.1.100:8045
>>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
>>> for, nodeId=kreator:46477, nodeAddresskreator:8042,
>>> nodeRackName/default-rack, nodeNumContainers7
>>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
>>> for, nodeId=exotica:46645, nodeAddressexotica:8042,
>>> nodeRackName/default-rack, nodeNumContainers8
>>> Exception in thread "main" java.lang.RuntimeException: Giraph job
>>> requires 2 containers to run; cluster only hosts 15
>>> at
>>> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230)
>>> at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:125)
>>> ....
>>>
>>> It seems like some config is missing from our hadoop setup.
>>> Also, note that I'm working from the trunk branch.
>>>
>>> Thanx,
>>> Vishal
>>>
>>>
>>> On Wed, Nov 27, 2013 at 11:34 AM, Vishal Mishra <mi...@gmail.com>wrote:
>>>
>>>> Hey Guys,
>>>> I'm trying giraph 1.0.0 on hadoop 2.2.0 (2-node cluster) but hitting
>>>> the same problem as outlined in this thread. Was there a resolution? If
>>>> yes, can you please share?
>>>> Completely new to giraph and hadoop so please excuse any omissions. Our
>>>> cluster is fully functional and running mapreduce and hive jobs cleanly.
>>>>
>>>> Command:
>>>> hadoop jar
>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner -D giraph.zkList="zkNode.net:2081"
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/hduser/rrdata/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/hduser/rrdata/output -w 1 -yj
>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>
>>>> Trimmed Console Output:
>>>> 13/11/27 11:19:58 INFO yarn.GiraphYarnClient:
>>>> appattempt_1385518875931_0002_000001, State: ACCEPTED, Containers used: 1
>>>> 13/11/27 11:20:00 ERROR yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>>> state, diagnostics show: Application application_1385518875931_0002 failed
>>>> 2 times due to AM Container for appattempt_1385518875931_0002_000002 exited
>>>> with exitCode: 1 due to: Exception from container-launch:
>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>> at
>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>>
>>>>
>>>> Thanx,
>>>> Vishal
>>>>
>>>>
>>>> On Tue, Nov 19, 2013 at 3:16 PM, Mohammad Islam <mi...@yahoo.com>wrote:
>>>>
>>>>> Rob,
>>>>> I understand your frustration.
>>>>> I'm pre-occupied with some other stuffs in next couple of days.
>>>>> If no one replied by then, I will recreate the situation here and try
>>>>> to give you more concrete help.
>>>>>
>>>>> Regards,
>>>>> Mohammad
>>>>>
>>>>>
>>>>> On Tuesday, November 19, 2013 2:56 AM, Rob Vesse <
>>>>> rvesse@dotnetrdf.org> wrote:
>>>>> Mohammad
>>>>>
>>>>> Comments inline:
>>>>>
>>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>>> Reply-To: <us...@giraph.apache.org>
>>>>> Date: Monday, 18 November 2013 22:24
>>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>>> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and
>>>>> Hadoop 2.2.0)
>>>>>
>>>>> I used the following shell script to run PR where my current working
>>>>> directory contains the graph*.jar in Hadoop 2.1.1.
>>>>> . You can try something similar.
>>>>>
>>>>>
>>>>> I already am.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> HAD_VERSION=2.1.1-SNAPSHOT
>>>>> hadoop fs -rmr out
>>>>> export GIRAPH_HOME=/home/mislam/softwares/giraph
>>>>> #export
>>>>> GIRAPH_CORE=$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>> #export
>>>>> GIRAPH_EXAMPLE=$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>
>>>>>
>>>>> CLASSPATH=$CLASSPATH:$GIRAPH_HOME/conf
>>>>>
>>>>>
>>>>> What is the point of putting the conf directory on the CLASSPATH? I
>>>>> assume this is only to pick up the giraph configuration but at the same
>>>>> time there is nothing to stop me putting my Giraph specific config in one
>>>>> of the other Hadoop configuration files e.g. core-site.xml?
>>>>>
>>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_CORE
>>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_EXAMPLE
>>>>>
>>>>> export CLASSPATH=$CLASSPATH
>>>>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
>>>>>
>>>>> echo $HADOOP_CLASSPATH
>>>>>
>>>>>
>>>>> I assume the point of the above is to ensure that the Giraph config
>>>>> file is on the class path Hadoop actually uses but adding this makes no
>>>>> difference to me, I get exactly the same failure as before. Modifying the
>>>>> logic to add the examples JAR onto the class path does not help either.
>>>>>
>>>>>
>>>>>
>>>>> hadoop --config $HADOOP_HOME/etc/hadoop jar
>>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>> org.apache.giraph.GiraphRunner
>>>>> org.apache.giraph.examples.SimplePageRankComputation -c
>>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 1 -yh 1024 -yj
>>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>> -vif
>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -mc
>>>>> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
>>>>> -vip input -op out
>>>>>
>>>>>
>>>>> This is not the same command that I am running, also DoubleSumCombiner
>>>>> gives a ClassNotFoundException, you actually want DoubleSumMessageCombiner.
>>>>> If I adapt my script to run your command I experience the exact same error
>>>>> as I do trying to run the command I showed so I am still nowhere.
>>>>>
>>>>> I'm getting a little frustrated that no-one has any idea what the
>>>>> problem is here, either this is a simple misconfiguration on my part or a
>>>>> bug in Giraph for Hadoop 2.2.0 ? Given the thread so far and the fact that
>>>>> I can run MR based Giraph on Hadoop 1.2.1 this is looking increasingly
>>>>> unlikely I'm starting to lean heavily towards the latter option.
>>>>>
>>>>> Rob
>>>>>
>>>>>
>>>>>
>>>>> On Monday, November 18, 2013 2:24 AM, Rob Vesse <
>>>>> rvesse@dotnetrdf.org> wrote:
>>>>> Hey all
>>>>>
>>>>> Bumping this thread because I am still completely blocked on this and
>>>>> hoping someone can help me. As explained earlier in the thread I have
>>>>> Giraph built for pure YARN on stable Hadoop 2.2.0 release like so:
>>>>>
>>>>> mvn –Phadoop_yarn –Dhadoop.version=2.2.0 –DskipTests=true clean install
>>>>>
>>>>> But when I try and run the simple shortest paths demo like so:
>>>>>
>>>>> $> bin/hadoop jar
>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>> org.apache.giraph.GiraphRunner
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>> /user/rvesse/shortestpaths -w 1 -yj
>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>
>>>>> The application fails and the YARN logs show the following error:
>>>>>
>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>
>>>>>
>>>>> Any ideas why the GiraphApplicationMaster would not be found? Have I just got something basic wrong in my configuration or is this a bug in Giraph?
>>>>>
>>>>>
>>>>> I found an earlier thread on the same topic (http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c51E5A4D1.40006@googlemail.com%3e) but it was never resolved hence why I have now changed the topic in the hope it will jog someone's memory
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>>
>>>>> Rob
>>>>>
>>>>>
>>>>> From: Rob Vesse <rv...@dotnetrdf.org>
>>>>> Reply-To: <us...@giraph.apache.org>
>>>>> Date: Friday, 15 November 2013 14:14
>>>>> To: <us...@giraph.apache.org>
>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>
>>>>> Adding the –yj argument to specify the YARN JARs doesn't make any
>>>>> difference.
>>>>>
>>>>> Here is the complete console output from my run:
>>>>>
>>>>> bin/hadoop jar
>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>> org.apache.giraph.GiraphRunner
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>> /user/rvesse/shortestpaths/3 -w 1 -yj
>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>> 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info from
>>>>> SCDynamicStore
>>>>> 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load
>>>>> native-hadoop library for your platform... using builtin-java classes where
>>>>> applicable
>>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format
>>>>> specified. Ensure your InputFormat does not require one.
>>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output format
>>>>> specified. Ensure your OutputFormat does not require one.
>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is:
>>>>> hdfs://localhost/user/rvesse/shortestpaths/3
>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
>>>>> 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager
>>>>> at /0.0.0.0:8032
>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from ASM
>>>>> for, nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042,
>>>>> nodeRackName/default-rack, nodeNumContainers0
>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task heap
>>>>> size to 1024MB.
>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application
>>>>> ID: application_1384523910015_0004
>>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is
>>>>> deprecated. Instead, use mapreduce.job.id
>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for
>>>>> the application master
>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM
>>>>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap
>>>>> ....
>>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir is
>>>>> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in
>>>>> LocalResources ::
>>>>> hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_1384523910015_0004/giraph-conf.xml
>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS
>>>>> :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>> /contrib/capacity-scheduler/*.jar
>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>> /contrib/capacity-scheduler
>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient:
>>>>> ApplicationSumbissionContext for GiraphApplicationMaster launch container
>>>>> is populated.
>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application
>>>>> to ASM
>>>>> 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application
>>>>> application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after
>>>>> submission :application_1384523910015_0004
>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>>>>> container request was submitted to ResourceManager for job: Giraph:
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation
>>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.82
>>>>> secs
>>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
>>>>> appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers used: 1
>>>>> 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>>>> state, diagnostics show: Application application_1384523910015_0004 failed
>>>>> 2 times due to AM Container for appattempt_1384523910015_0004_000002 exited
>>>>> with exitCode: 1 due to: Exception from container-launch:
>>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>>> at
>>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>>>> at
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>> at java.lang.Thread.run(Thread.java:695)
>>>>>
>>>>>
>>>>> .Failing this attempt.. Failing the application.
>>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS
>>>>> distributed cache directory for Giraph job.
>>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
>>>>> running time: 0 minutes, 3 seconds.
>>>>>
>>>>> And I see the same error in the logs:
>>>>>
>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>
>>>>>
>>>>> Any more ideas?
>>>>>
>>>>>
>>>>> Rob
>>>>>
>>>>>
>>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>>> Reply-To: <us...@giraph.apache.org>
>>>>> Date: Thursday, 14 November 2013 20:36
>>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>
>>>>> I tried the following command few months back. I found you are not
>>>>> passing some params.
>>>>>
>>>>> bin/hadoop --config etc/hadoop jar
>>>>> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-examples-1.?.0-for-hadoop-???-jar-with-dependencies.jar
>>>>> org.apache.giraph.GiraphRunner
>>>>> org.apache.giraph.examples.SimplePageRankVertexOpt -c
>>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj
>>>>> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif
>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -of
>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip
>>>>> /user/me/input -op /user/me/output_35w_20G_80m_1
>>>>>
>>>>>
>>>>> On Thursday, November 14, 2013 3:22 AM, Rob Vesse <
>>>>> rvesse@dotnetrdf.org> wrote:
>>>>> It's the same command line I showed earlier, just trying to run the
>>>>> example application:
>>>>>
>>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>> org.apache.giraph.GiraphRunner
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>> /user/rvesse/shortestpaths/2 -w 1
>>>>>
>>>>> Rob
>>>>>
>>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>>> Reply-To: <us...@giraph.apache.org>
>>>>> Date: Thursday, 14 November 2013 05:57
>>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>
>>>>> What are you running? PR?
>>>>> What is the command line?
>>>>>
>>>>>
>>>>>
>>>>> On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <
>>>>> rvesse@dotnetrdf.org> wrote:
>>>>> So I fixed the previous issue I was having which I reported and
>>>>> submitted a patch as GIRAPH-795
>>>>>
>>>>> I then went ahead with installing and configuring my external
>>>>> Zookeeper appropriately and tried running again but now the YARN
>>>>> application fails and I find the following error in the logs:
>>>>>
>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>> org/apache/giraph/yarn/GiraphApplicationMaster
>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>
>>>>>
>>>>> Anybody have idea what the cause of this one might be or how to fix it?
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>>
>>>>> Rob
>>>>>
>>>>>
>>>>> From: Rob Vesse <rv...@dotnetrdf.org>
>>>>> Reply-To: <us...@giraph.apache.org>
>>>>> Date: Wednesday, 13 November 2013 15:35
>>>>> To: <us...@giraph.apache.org>
>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>
>>>>> Mohammad
>>>>>
>>>>> This got me a build that at least attempts to start to run but not I
>>>>> see the following:
>>>>>
>>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>> org.apache.giraph.GiraphRunner
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>> /user/rvesse/shortestpaths/2 -w 1
>>>>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info
>>>>> from SCDynamicStore
>>>>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load
>>>>> native-hadoop library for your platform... using builtin-java classes where
>>>>> applicable
>>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format
>>>>> specified. Ensure your InputFormat does not require one.
>>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output format
>>>>> specified. Ensure your OutputFormat does not require one.
>>>>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is:
>>>>> hdfs://localhost/user/rvesse/shortestpaths/2
>>>>> Exception in thread "main" java.lang.NullPointerException
>>>>> at
>>>>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(GiraphYarnClient.java:460)
>>>>> at
>>>>> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>>
>>>>> Which indirectly implies that I have some missing configuration to run
>>>>> Giraph on YARN. I don't think that should be a NPE and it looks to be a
>>>>> trivial fix so I'll try and fix and send a pull request for that.
>>>>>
>>>>> The error message I should receive if the NPE wasn't happening is the
>>>>> following:
>>>>>
>>>>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances:
>>>>> use a standalone ZooKeeper.
>>>>>
>>>>> What configuration options do I need to have set to point Giraph at an
>>>>> external Zookeeper?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Rob
>>>>>
>>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>>> Reply-To: <us...@giraph.apache.org>
>>>>> Date: Wednesday, 13 November 2013 11:15
>>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>
>>>>> Hi Rob,
>>>>> Did you try with this command "mvn -Phadoop_yarn
>>>>> -Dhadoop.version=2.2.0 clean package -DskipTests"?
>>>>> Regards,
>>>>> Mohammad
>>>>>
>>>>>
>>>>> On Wednesday, November 13, 2013 2:55 AM, Rob Vesse <
>>>>> rvesse@dotnetrdf.org> wrote:
>>>>> Hi all
>>>>>
>>>>> I'm having trouble getting Giraph to build and run against a Hadoop
>>>>> 2.2.0 installation. I tried the hadoop_2.0.3 profile with no luck because
>>>>> I get class cast errors when trying to run the examples.
>>>>>
>>>>> Is there a specific profile I should be building to support Hadoop
>>>>> 2.2.0 or is this not supported yet?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Rob
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)
Posted by Rob Vesse <rv...@dotnetrdf.org>.
Yes it actually is in the JAR file:
mbp-rvesse:target rvesse$ jar -tf
/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examp
les-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar | grep
org.apache.giraph.yarn.GiraphApplicationMaster
org/apache/giraph/yarn/GiraphApplicationMaster$1.class
org/apache/giraph/yarn/GiraphApplicationMaster$LaunchContainerRunnable.class
org/apache/giraph/yarn/GiraphApplicationMaster$NMCallbackHandler.class
org/apache/giraph/yarn/GiraphApplicationMaster$RMCallbackHandler.class
org/apache/giraph/yarn/GiraphApplicationMaster.class
Rob
From: Rafal Wojdyla <ra...@gmail.com>
Reply-To: <us...@giraph.apache.org>
Date: Monday, 9 December 2013 10:24
To: <us...@giraph.apache.org>
Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop
2.2.0)
> Hi all!
>
> I had the same problems as you guys:
>
> @Rob:
> Did you check if org.apache.giraph.yarn.GiraphApplicationMaster is actually in
> the jar file - it wasn't it my case - I'm using custom profile in maven.
> Do: jar -tf
> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-example
> s-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar | grep
> org.apache.giraph.yarn.GiraphApplicationMaster
>
> @Vishal:
> I got this error few times when cluster was highly utilized, problem
> disappeared when I used queue with free resources - this may be a problem but
> I haven't investigated it.
>
> Unfortunately I wasn't able to successfully run Giraph application anyway, due
> to: https://issues.apache.org/jira/browse/GIRAPH-806
>
> Cheers - Rafal
>
>
> On Sun, Dec 8, 2013 at 7:23 PM, Eli Reisman <ap...@gmail.com> wrote:
>> Hi all,
>>
>> It looks to me like you want to run the giraph-core jar at the beginning of
>> the "hadoop jar" call and supply the giraph-examples jar with -yj argument.
>>
>> Also, you can place any -D type configuration settings you're sick of
>> supplying at the command line into giraph-site.xml just as you would with
>> Hadoop settings in one of its own site.xml files, and place these where your
>> local Hadoop client configs are and they should be picked up in your
>> application.
>>
>> Hope that helps,
>>
>> Eli
>>
>>
>>
>> On Thu, Nov 28, 2013 at 12:21 PM, Vishal Mishra <mi...@gmail.com>
>> wrote:
>>> I had a mistake in my command (moronic copy/paste zookeeper settings error).
>>> Updated and now hitting the following:
>>>
>>> ....
>>> 13/11/28 12:02:14 INFO yarn.GiraphYarnClient: Running Client
>>> 13/11/28 12:02:14 INFO client.RMProxy: Connecting to ResourceManager at
>>> master/192.168.1.100:8045 <http://192.168.1.100:8045>
>>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM for,
>>> nodeId=kreator:46477, nodeAddresskreator:8042, nodeRackName/default-rack,
>>> nodeNumContainers7
>>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM for,
>>> nodeId=exotica:46645, nodeAddressexotica:8042, nodeRackName/default-rack,
>>> nodeNumContainers8
>>> Exception in thread "main" java.lang.RuntimeException: Giraph job requires 2
>>> containers to run; cluster only hosts 15
>>> at
>>> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(Girap
>>> hYarnClient.java:230)
>>> at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:125)
>>> ....
>>>
>>> It seems like some config is missing from our hadoop setup.
>>> Also, note that I'm working from the trunk branch.
>>>
>>> Thanx,
>>> Vishal
>>>
>>>
>>> On Wed, Nov 27, 2013 at 11:34 AM, Vishal Mishra <mi...@gmail.com>
>>> wrote:
>>>> Hey Guys,
>>>> I'm trying giraph 1.0.0 on hadoop 2.2.0 (2-node cluster) but hitting the
>>>> same problem as outlined in this thread. Was there a resolution? If yes,
>>>> can you please share?
>>>> Completely new to giraph and hadoop so please excuse any omissions. Our
>>>> cluster is fully functional and running mapreduce and hive jobs cleanly.
>>>>
>>>> Command:
>>>> hadoop jar
>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hado
>>>> op-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner -D
>>>> giraph.zkList="zkNode.net:2081"
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/hduser/rrdata/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/hduser/rrdata/output -w 1 -yj
>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hado
>>>> op-2.2.0-jar-with-dependencies.jar
>>>>
>>>> Trimmed Console Output:
>>>> 13/11/27 11:19:58 INFO yarn.GiraphYarnClient:
>>>> appattempt_1385518875931_0002_000001, State: ACCEPTED, Containers used: 1
>>>> 13/11/27 11:20:00 ERROR yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>>> state, diagnostics show: Application application_1385518875931_0002 failed
>>>> 2 times due to AM Container for appattempt_1385518875931_0002_000002 exited
>>>> with exitCode: 1 due to: Exception from container-launch:
>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>> at
>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchCo
>>>> ntainer(DefaultContainerExecutor.java:195)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contain
>>>> erLaunch.call(ContainerLaunch.java:283)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contain
>>>> erLaunch.call(ContainerLaunch.java:79)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1
>>>> 145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
>>>> 615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>>
>>>>
>>>> Thanx,
>>>> Vishal
>>>>
>>>>
>>>> On Tue, Nov 19, 2013 at 3:16 PM, Mohammad Islam <mi...@yahoo.com> wrote:
>>>>> Rob,
>>>>> I understand your frustration.
>>>>> I'm pre-occupied with some other stuffs in next couple of days.
>>>>> If no one replied by then, I will recreate the situation here and try to
>>>>> give you more concrete help.
>>>>>
>>>>> Regards,
>>>>> Mohammad
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tuesday, November 19, 2013 2:56 AM, Rob Vesse <rv...@dotnetrdf.org>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Mohammad
>>>>>
>>>>> Comments inline:
>>>>>
>>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>>> Reply-To: <us...@giraph.apache.org>
>>>>> Date: Monday, 18 November 2013 22:24
>>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>>> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop
>>>>> 2.2.0)
>>>>>
>>>>>> I used the following shell script to run PR where my current working
>>>>>> directory contains the graph*.jar in Hadoop 2.1.1.
>>>>>> . You can try something similar.
>>>>>
>>>>> I already am.
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> HAD_VERSION=2.1.1-SNAPSHOT
>>>>>> hadoop fs -rmr out
>>>>>> export GIRAPH_HOME=/home/mislam/softwares/giraph
>>>>>> #export
>>>>>> GIRAPH_CORE=$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-had
>>>>>> oop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>> #export
>>>>>> GIRAPH_EXAMPLE=$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-
>>>>>> SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>>>
>>>>>>
>>>>>> CLASSPATH=$CLASSPATH:$GIRAPH_HOME/conf
>>>>>
>>>>> What is the point of putting the conf directory on the CLASSPATH? I
>>>>> assume this is only to pick up the giraph configuration but at the same
>>>>> time there is nothing to stop me putting my Giraph specific config in one
>>>>> of the other Hadoop configuration files e.g. core-site.xml?
>>>>>
>>>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_CORE
>>>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_EXAMPLE
>>>>>>
>>>>>> export CLASSPATH=$CLASSPATH
>>>>>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
>>>>>>
>>>>>> echo $HADOOP_CLASSPATH
>>>>>
>>>>> I assume the point of the above is to ensure that the Giraph config file
>>>>> is on the class path Hadoop actually uses but adding this makes no
>>>>> difference to me, I get exactly the same failure as before. Modifying the
>>>>> logic to add the examples JAR onto the class path does not help either.
>>>>>
>>>>>>
>>>>>>
>>>>>> hadoop --config $HADOOP_HOME/etc/hadoop jar
>>>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependenc
>>>>>> ies.jar org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimplePageRankComputation -c
>>>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 1 -yh 1024 -yj
>>>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependenc
>>>>>> ies.jar -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -mc
>>>>>> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMaste
>>>>>> rCompute -vip input -op out
>>>>>
>>>>> This is not the same command that I am running, also DoubleSumCombiner
>>>>> gives a ClassNotFoundException, you actually want
>>>>> DoubleSumMessageCombiner. If I adapt my script to run your command I
>>>>> experience the exact same error as I do trying to run the command I showed
>>>>> so I am still nowhere.
>>>>>
>>>>> I'm getting a little frustrated that no-one has any idea what the problem
>>>>> is here, either this is a simple misconfiguration on my part or a bug in
>>>>> Giraph for Hadoop 2.2.0 ? Given the thread so far and the fact that I can
>>>>> run MR based Giraph on Hadoop 1.2.1 this is looking increasingly unlikely
>>>>> I'm starting to lean heavily towards the latter option.
>>>>>
>>>>> Rob
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Monday, November 18, 2013 2:24 AM, Rob Vesse <rv...@dotnetrdf.org>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> Hey all
>>>>>>
>>>>>> Bumping this thread because I am still completely blocked on this and
>>>>>> hoping someone can help me. As explained earlier in the thread I have
>>>>>> Giraph built for pure YARN on stable Hadoop 2.2.0 release like so:
>>>>>>
>>>>>> mvn Phadoop_yarn Dhadoop.version=2.2.0 DskipTests=true clean install
>>>>>>
>>>>>> But when I try and run the simple shortest paths demo like so:
>>>>>>
>>>>>> $> bin/hadoop jar
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-ex
>>>>>> amples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>> /user/rvesse/shortestpaths -w 1 -yj
>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-ex
>>>>>> amples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>
>>>>>> The application fails and the YARN logs show the following error:
>>>>>>
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>> org/apache/giraph/yarn/GiraphApplicationMaster
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>
>>>>>> Any ideas why the GiraphApplicationMaster would not be found? Have I
>>>>>> just got something basic wrong in my configuration or is this a bug in
>>>>>> Giraph?
>>>>>>
>>>>>> I found an earlier thread on the same topic
>>>>>> (http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c51E5
>>>>>> A4D1.40006@googlemail.com%3e) but it was never resolved hence why I have
>>>>>> now changed the topic in the hope it will jog someone's memory
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>> From: Rob Vesse <rv...@dotnetrdf.org>
>>>>>> Reply-To: <us...@giraph.apache.org>
>>>>>> Date: Friday, 15 November 2013 14:14
>>>>>> To: <us...@giraph.apache.org>
>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>
>>>>>>> Adding the yj argument to specify the YARN JARs doesn't make any
>>>>>>> difference.
>>>>>>>
>>>>>>> Here is the complete console output from my run:
>>>>>>>
>>>>>>> bin/hadoop jar
>>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-e
>>>>>>> xamples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>> org.apache.giraph.GiraphRunner
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>>> /user/rvesse/shortestpaths/3 -w 1 -yj
>>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-e
>>>>>>> xamples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>> 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info from
>>>>>>> SCDynamicStore
>>>>>>> 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load
>>>>>>> native-hadoop library for your platform... using builtin-java classes
>>>>>>> where applicable
>>>>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format
>>>>>>> specified. Ensure your InputFormat does not require one.
>>>>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output format
>>>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is:
>>>>>>> hdfs://localhost/user/rvesse/shortestpaths/3
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
>>>>>>> 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager at
>>>>>>> /0.0.0.0:8032 <http://0.0.0.0:8032>
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from ASM
>>>>>>> for, nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042,
>>>>>>> nodeRackName/default-rack, nodeNumContainers0
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task heap
>>>>>>> size to 1024MB.
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application
>>>>>>> ID: application_1384523910015_0004
>>>>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id
>>>>>>> <http://mapred.job.id> is deprecated. Instead, use mapreduce.job.id
>>>>>>> <http://mapreduce.job.id>
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for
>>>>>>> the application master
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM
>>>>>>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/
>>>>>>> hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HD
>>>>>>> FS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$H
>>>>>>> ADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/
>>>>>>> lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/s
>>>>>>> hare/hadoop/mapreduce/lib/*}
>>>>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>>>>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir is
>>>>>>> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in LocalResources
>>>>>>> ::
>>>>>>> hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_138452391
>>>>>>> 0015_0004/giraph-conf.xml
>>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS
>>>>>>> :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-
>>>>>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>>>>> /contrib/capacity-scheduler/*.jar
>>>>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>>>>> /contrib/capacity-scheduler
>>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient:
>>>>>>> ApplicationSumbissionContext for GiraphApplicationMaster launch
>>>>>>> container is populated.
>>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application to
>>>>>>> ASM
>>>>>>> 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application
>>>>>>> application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
>>>>>>> <http://0.0.0.0:8032>
>>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after
>>>>>>> submission :application_1384523910015_0004
>>>>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>>>>>>> container request was submitted to ResourceManager for job: Giraph:
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation
>>>>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.82
>>>>>>> secs
>>>>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
>>>>>>> appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers used:
>>>>>>> 1
>>>>>>> 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>>>>>> state, diagnostics show: Application application_1384523910015_0004
>>>>>>> failed 2 times due to AM Container for
>>>>>>> appattempt_1384523910015_0004_000002 exited with exitCode: 1 due to:
>>>>>>> Exception from container-launch:
>>>>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>>>>> at
>>>>>>>
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589>>>>>>>
)
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launc
>>>>>>> hContainer(DefaultContainerExecutor.java:195)
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont
>>>>>>> ainerLaunch.call(ContainerLaunch.java:283)
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont
>>>>>>> ainerLaunch.call(ContainerLaunch.java:79)
>>>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
>>>>>>> r.java:895)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>>>>> va:918)
>>>>>>> at java.lang.Thread.run(Thread.java:695)
>>>>>>>
>>>>>>>
>>>>>>> .Failing this attempt.. Failing the application.
>>>>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS
>>>>>>> distributed cache directory for Giraph job.
>>>>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
>>>>>>> running time: 0 minutes, 3 seconds.
>>>>>>>
>>>>>>> And I see the same error in the logs:
>>>>>>>
>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>> org/apache/giraph/yarn/GiraphApplicationMaster
>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>> org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>>
>>>>>>> Any more ideas?
>>>>>>>
>>>>>>> Rob
>>>>>>>
>>>>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>>>>> Reply-To: <us...@giraph.apache.org>
>>>>>>> Date: Thursday, 14 November 2013 20:36
>>>>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>>
>>>>>>> I tried the following command few months back. I found you are not
>>>>>>> passing some params.
>>>>>>>
>>>>>>> bin/hadoop --config etc/hadoop jar
>>>>>>> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-exam
>>>>>>> ples-1.?.0-for-hadoop-???-jar-with-dependencies.jar
>>>>>>> org.apache.giraph.GiraphRunner
>>>>>>> org.apache.giraph.examples.SimplePageRankVertexOpt -c
>>>>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj
>>>>>>> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif
>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>> -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip
>>>>>>> /user/me/input -op /user/me/output_35w_20G_80m_1
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thursday, November 14, 2013 3:22 AM, Rob Vesse
>>>>>>> <rv...@dotnetrdf.org> wrote:
>>>>>>>
>>>>>>>
>>>>>>> It's the same command line I showed earlier, just trying to run the
>>>>>>> example application:
>>>>>>>
>>>>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-e
>>>>>>> xamples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>> org.apache.giraph.GiraphRunner
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>>> /user/rvesse/shortestpaths/2 -w 1
>>>>>>>
>>>>>>> Rob
>>>>>>>
>>>>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>>>>> Reply-To: <us...@giraph.apache.org>
>>>>>>> Date: Thursday, 14 November 2013 05:57
>>>>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>>
>>>>>>> What are you running? PR?
>>>>>>> What is the command line?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wednesday, November 13, 2013 8:30 AM, Rob Vesse
>>>>>>> <rv...@dotnetrdf.org> wrote:
>>>>>>>
>>>>>>>
>>>>>>> So I fixed the previous issue I was having which I reported and
>>>>>>> submitted a patch as GIRAPH-795
>>>>>>>
>>>>>>> I then went ahead with installing and configuring my external Zookeeper
>>>>>>> appropriately and tried running again but now the YARN application fails
>>>>>>> and I find the following error in the logs:
>>>>>>>
>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>> org/apache/giraph/yarn/GiraphApplicationMaster
>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>> org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>>
>>>>>>> Anybody have idea what the cause of this one might be or how to fix it?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Rob
>>>>>>>
>>>>>>> From: Rob Vesse <rv...@dotnetrdf.org>
>>>>>>> Reply-To: <us...@giraph.apache.org>
>>>>>>> Date: Wednesday, 13 November 2013 15:35
>>>>>>> To: <us...@giraph.apache.org>
>>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>>
>>>>>>> Mohammad
>>>>>>>
>>>>>>> This got me a build that at least attempts to start to run but not I see
>>>>>>> the following:
>>>>>>>
>>>>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-e
>>>>>>> xamples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>>>> org.apache.giraph.GiraphRunner
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>>> /user/rvesse/shortestpaths/2 -w 1
>>>>>>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info from
>>>>>>> SCDynamicStore
>>>>>>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load
>>>>>>> native-hadoop library for your platform... using builtin-java classes
>>>>>>> where applicable
>>>>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format
>>>>>>> specified. Ensure your InputFormat does not require one.
>>>>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output format
>>>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is:
>>>>>>> hdfs://localhost/user/rvesse/shortestpaths/2
>>>>>>> Exception in thread "main" java.lang.NullPointerException
>>>>>>> at
>>>>>>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(
>>>>>>> GiraphYarnClient.java:460)
>>>>>>> at
>>>>>>> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
>>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
>>>>>>> a:39)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>>>>> Impl.java:25)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>>>>
>>>>>>> Which indirectly implies that I have some missing configuration to run
>>>>>>> Giraph on YARN. I don't think that should be a NPE and it looks to be a
>>>>>>> trivial fix so I'll try and fix and send a pull request for that.
>>>>>>>
>>>>>>> The error message I should receive if the NPE wasn't happening is the
>>>>>>> following:
>>>>>>>
>>>>>>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances:
>>>>>>> use a standalone ZooKeeper.
>>>>>>>
>>>>>>> What configuration options do I need to have set to point Giraph at an
>>>>>>> external Zookeeper?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Rob
>>>>>>>
>>>>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>>>>> Reply-To: <us...@giraph.apache.org>
>>>>>>> Date: Wednesday, 13 November 2013 11:15
>>>>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>>>>
>>>>>>> Hi Rob,
>>>>>>> Did you try with this command "mvn -Phadoop_yarn -Dhadoop.version=2.2.0
>>>>>>> clean package -DskipTests"?
>>>>>>> Regards,
>>>>>>> Mohammad
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wednesday, November 13, 2013 2:55 AM, Rob Vesse
>>>>>>> <rv...@dotnetrdf.org> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi all
>>>>>>>
>>>>>>> I'm having trouble getting Giraph to build and run against a Hadoop
>>>>>>> 2.2.0 installation. I tried the hadoop_2.0.3 profile with no luck
>>>>>>> because I get class cast errors when trying to run the examples.
>>>>>>>
>>>>>>> Is there a specific profile I should be building to support Hadoop 2.2.0
>>>>>>> or is this not supported yet?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Rob
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)
Posted by Rafal Wojdyla <ra...@gmail.com>.
Hi all!
I had the same problems as you guys:
@Rob:
Did you check if *org.apache.giraph.yarn.GiraphApplicationMaster* is actually
in the jar file - it wasn't it my case - I'm using custom profile in maven.
Do: *jar -tf
/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
| grep org.apache.giraph.yarn.GiraphApplicationMaster*
@Vishal:
I got this error few times when cluster was highly utilized, problem
disappeared when I used queue with free resources - this may be a problem
but I haven't investigated it.
Unfortunately I wasn't able to successfully run Giraph application anyway,
due to: https://issues.apache.org/jira/browse/GIRAPH-806
Cheers - Rafal
On Sun, Dec 8, 2013 at 7:23 PM, Eli Reisman <ap...@gmail.com>wrote:
> Hi all,
>
> It looks to me like you want to run the giraph-core jar at the beginning
> of the "hadoop jar" call and supply the giraph-examples jar with -yj
> argument.
>
> Also, you can place any -D type configuration settings you're sick of
> supplying at the command line into giraph-site.xml just as you would with
> Hadoop settings in one of its own site.xml files, and place these where
> your local Hadoop client configs are and they should be picked up in your
> application.
>
> Hope that helps,
>
> Eli
>
>
>
> On Thu, Nov 28, 2013 at 12:21 PM, Vishal Mishra <mi...@gmail.com>wrote:
>
>> I had a mistake in my command (moronic copy/paste zookeeper settings
>> error).
>> Updated and now hitting the following:
>>
>> ....
>> 13/11/28 12:02:14 INFO yarn.GiraphYarnClient: Running Client
>> 13/11/28 12:02:14 INFO client.RMProxy: Connecting to ResourceManager at
>> master/192.168.1.100:8045
>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
>> for, nodeId=kreator:46477, nodeAddresskreator:8042,
>> nodeRackName/default-rack, nodeNumContainers7
>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
>> for, nodeId=exotica:46645, nodeAddressexotica:8042,
>> nodeRackName/default-rack, nodeNumContainers8
>> Exception in thread "main" java.lang.RuntimeException: Giraph job
>> requires 2 containers to run; cluster only hosts 15
>> at
>> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230)
>> at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:125)
>> ....
>>
>> It seems like some config is missing from our hadoop setup.
>> Also, note that I'm working from the trunk branch.
>>
>> Thanx,
>> Vishal
>>
>>
>> On Wed, Nov 27, 2013 at 11:34 AM, Vishal Mishra <mi...@gmail.com>wrote:
>>
>>> Hey Guys,
>>> I'm trying giraph 1.0.0 on hadoop 2.2.0 (2-node cluster) but hitting the
>>> same problem as outlined in this thread. Was there a resolution? If yes,
>>> can you please share?
>>> Completely new to giraph and hadoop so please excuse any omissions. Our
>>> cluster is fully functional and running mapreduce and hive jobs cleanly.
>>>
>>> Command:
>>> hadoop jar
>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>> org.apache.giraph.GiraphRunner -D giraph.zkList="zkNode.net:2081"
>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>> -vip /user/hduser/rrdata/tiny_graph.txt -vof
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>> /user/hduser/rrdata/output -w 1 -yj
>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>
>>> Trimmed Console Output:
>>> 13/11/27 11:19:58 INFO yarn.GiraphYarnClient:
>>> appattempt_1385518875931_0002_000001, State: ACCEPTED, Containers used: 1
>>> 13/11/27 11:20:00 ERROR yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>> state, diagnostics show: Application application_1385518875931_0002 failed
>>> 2 times due to AM Container for appattempt_1385518875931_0002_000002 exited
>>> with exitCode: 1 due to: Exception from container-launch:
>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>> at
>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>>
>>>
>>> Thanx,
>>> Vishal
>>>
>>>
>>> On Tue, Nov 19, 2013 at 3:16 PM, Mohammad Islam <mi...@yahoo.com>wrote:
>>>
>>>> Rob,
>>>> I understand your frustration.
>>>> I'm pre-occupied with some other stuffs in next couple of days.
>>>> If no one replied by then, I will recreate the situation here and try
>>>> to give you more concrete help.
>>>>
>>>> Regards,
>>>> Mohammad
>>>>
>>>>
>>>> On Tuesday, November 19, 2013 2:56 AM, Rob Vesse <
>>>> rvesse@dotnetrdf.org> wrote:
>>>> Mohammad
>>>>
>>>> Comments inline:
>>>>
>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>> Reply-To: <us...@giraph.apache.org>
>>>> Date: Monday, 18 November 2013 22:24
>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop
>>>> 2.2.0)
>>>>
>>>> I used the following shell script to run PR where my current working
>>>> directory contains the graph*.jar in Hadoop 2.1.1.
>>>> . You can try something similar.
>>>>
>>>>
>>>> I already am.
>>>>
>>>>
>>>>
>>>>
>>>> HAD_VERSION=2.1.1-SNAPSHOT
>>>> hadoop fs -rmr out
>>>> export GIRAPH_HOME=/home/mislam/softwares/giraph
>>>> #export
>>>> GIRAPH_CORE=$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>> #export
>>>> GIRAPH_EXAMPLE=$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>
>>>>
>>>> CLASSPATH=$CLASSPATH:$GIRAPH_HOME/conf
>>>>
>>>>
>>>> What is the point of putting the conf directory on the CLASSPATH? I
>>>> assume this is only to pick up the giraph configuration but at the same
>>>> time there is nothing to stop me putting my Giraph specific config in one
>>>> of the other Hadoop configuration files e.g. core-site.xml?
>>>>
>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_CORE
>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_EXAMPLE
>>>>
>>>> export CLASSPATH=$CLASSPATH
>>>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
>>>>
>>>> echo $HADOOP_CLASSPATH
>>>>
>>>>
>>>> I assume the point of the above is to ensure that the Giraph config
>>>> file is on the class path Hadoop actually uses but adding this makes no
>>>> difference to me, I get exactly the same failure as before. Modifying the
>>>> logic to add the examples JAR onto the class path does not help either.
>>>>
>>>>
>>>>
>>>> hadoop --config $HADOOP_HOME/etc/hadoop jar
>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimplePageRankComputation -c
>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 1 -yh 1024 -yj
>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>> -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -mc
>>>> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
>>>> -vip input -op out
>>>>
>>>>
>>>> This is not the same command that I am running, also DoubleSumCombiner
>>>> gives a ClassNotFoundException, you actually want DoubleSumMessageCombiner.
>>>> If I adapt my script to run your command I experience the exact same error
>>>> as I do trying to run the command I showed so I am still nowhere.
>>>>
>>>> I'm getting a little frustrated that no-one has any idea what the
>>>> problem is here, either this is a simple misconfiguration on my part or a
>>>> bug in Giraph for Hadoop 2.2.0 ? Given the thread so far and the fact that
>>>> I can run MR based Giraph on Hadoop 1.2.1 this is looking increasingly
>>>> unlikely I'm starting to lean heavily towards the latter option.
>>>>
>>>> Rob
>>>>
>>>>
>>>>
>>>> On Monday, November 18, 2013 2:24 AM, Rob Vesse <rv...@dotnetrdf.org>
>>>> wrote:
>>>> Hey all
>>>>
>>>> Bumping this thread because I am still completely blocked on this and
>>>> hoping someone can help me. As explained earlier in the thread I have
>>>> Giraph built for pure YARN on stable Hadoop 2.2.0 release like so:
>>>>
>>>> mvn –Phadoop_yarn –Dhadoop.version=2.2.0 –DskipTests=true clean install
>>>>
>>>> But when I try and run the simple shortest paths demo like so:
>>>>
>>>> $> bin/hadoop jar
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/rvesse/shortestpaths -w 1 -yj
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>
>>>> The application fails and the YARN logs show the following error:
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Any ideas why the GiraphApplicationMaster would not be found? Have I just got something basic wrong in my configuration or is this a bug in Giraph?
>>>>
>>>>
>>>> I found an earlier thread on the same topic (http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c51E5A4D1.40006@googlemail.com%3e) but it was never resolved hence why I have now changed the topic in the hope it will jog someone's memory
>>>>
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> Rob
>>>>
>>>>
>>>> From: Rob Vesse <rv...@dotnetrdf.org>
>>>> Reply-To: <us...@giraph.apache.org>
>>>> Date: Friday, 15 November 2013 14:14
>>>> To: <us...@giraph.apache.org>
>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>
>>>> Adding the –yj argument to specify the YARN JARs doesn't make any
>>>> difference.
>>>>
>>>> Here is the complete console output from my run:
>>>>
>>>> bin/hadoop jar
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/rvesse/shortestpaths/3 -w 1 -yj
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info from
>>>> SCDynamicStore
>>>> 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load
>>>> native-hadoop library for your platform... using builtin-java classes where
>>>> applicable
>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format
>>>> specified. Ensure your InputFormat does not require one.
>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output format
>>>> specified. Ensure your OutputFormat does not require one.
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is:
>>>> hdfs://localhost/user/rvesse/shortestpaths/3
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
>>>> 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager at
>>>> /0.0.0.0:8032
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from ASM
>>>> for, nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042,
>>>> nodeRackName/default-rack, nodeNumContainers0
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task heap
>>>> size to 1024MB.
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application
>>>> ID: application_1384523910015_0004
>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is
>>>> deprecated. Instead, use mapreduce.job.id
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for
>>>> the application master
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM
>>>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir is
>>>> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in
>>>> LocalResources ::
>>>> hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_1384523910015_0004/giraph-conf.xml
>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS
>>>> :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /contrib/capacity-scheduler/*.jar
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /contrib/capacity-scheduler
>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient:
>>>> ApplicationSumbissionContext for GiraphApplicationMaster launch container
>>>> is populated.
>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application to
>>>> ASM
>>>> 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application
>>>> application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after
>>>> submission :application_1384523910015_0004
>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>>>> container request was submitted to ResourceManager for job: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation
>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.82
>>>> secs
>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
>>>> appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers used: 1
>>>> 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>>> state, diagnostics show: Application application_1384523910015_0004 failed
>>>> 2 times due to AM Container for appattempt_1384523910015_0004_000002 exited
>>>> with exitCode: 1 due to: Exception from container-launch:
>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>> at
>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>> at java.lang.Thread.run(Thread.java:695)
>>>>
>>>>
>>>> .Failing this attempt.. Failing the application.
>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS
>>>> distributed cache directory for Giraph job.
>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
>>>> running time: 0 minutes, 3 seconds.
>>>>
>>>> And I see the same error in the logs:
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>
>>>>
>>>> Any more ideas?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Rob
>>>>
>>>>
>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>> Reply-To: <us...@giraph.apache.org>
>>>> Date: Thursday, 14 November 2013 20:36
>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>
>>>> I tried the following command few months back. I found you are not
>>>> passing some params.
>>>>
>>>> bin/hadoop --config etc/hadoop jar
>>>> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-examples-1.?.0-for-hadoop-???-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimplePageRankVertexOpt -c
>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj
>>>> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -of
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip
>>>> /user/me/input -op /user/me/output_35w_20G_80m_1
>>>>
>>>>
>>>> On Thursday, November 14, 2013 3:22 AM, Rob Vesse <
>>>> rvesse@dotnetrdf.org> wrote:
>>>> It's the same command line I showed earlier, just trying to run the
>>>> example application:
>>>>
>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/rvesse/shortestpaths/2 -w 1
>>>>
>>>> Rob
>>>>
>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>> Reply-To: <us...@giraph.apache.org>
>>>> Date: Thursday, 14 November 2013 05:57
>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>
>>>> What are you running? PR?
>>>> What is the command line?
>>>>
>>>>
>>>>
>>>> On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <
>>>> rvesse@dotnetrdf.org> wrote:
>>>> So I fixed the previous issue I was having which I reported and
>>>> submitted a patch as GIRAPH-795
>>>>
>>>> I then went ahead with installing and configuring my external Zookeeper
>>>> appropriately and tried running again but now the YARN application fails
>>>> and I find the following error in the logs:
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> org/apache/giraph/yarn/GiraphApplicationMaster
>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>
>>>>
>>>> Anybody have idea what the cause of this one might be or how to fix it?
>>>>
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> Rob
>>>>
>>>>
>>>> From: Rob Vesse <rv...@dotnetrdf.org>
>>>> Reply-To: <us...@giraph.apache.org>
>>>> Date: Wednesday, 13 November 2013 15:35
>>>> To: <us...@giraph.apache.org>
>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>
>>>> Mohammad
>>>>
>>>> This got me a build that at least attempts to start to run but not I
>>>> see the following:
>>>>
>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/rvesse/shortestpaths/2 -w 1
>>>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info from
>>>> SCDynamicStore
>>>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load
>>>> native-hadoop library for your platform... using builtin-java classes where
>>>> applicable
>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format
>>>> specified. Ensure your InputFormat does not require one.
>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output format
>>>> specified. Ensure your OutputFormat does not require one.
>>>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is:
>>>> hdfs://localhost/user/rvesse/shortestpaths/2
>>>> Exception in thread "main" java.lang.NullPointerException
>>>> at
>>>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(GiraphYarnClient.java:460)
>>>> at
>>>> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>
>>>> Which indirectly implies that I have some missing configuration to run
>>>> Giraph on YARN. I don't think that should be a NPE and it looks to be a
>>>> trivial fix so I'll try and fix and send a pull request for that.
>>>>
>>>> The error message I should receive if the NPE wasn't happening is the
>>>> following:
>>>>
>>>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances:
>>>> use a standalone ZooKeeper.
>>>>
>>>> What configuration options do I need to have set to point Giraph at an
>>>> external Zookeeper?
>>>>
>>>> Cheers,
>>>>
>>>> Rob
>>>>
>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>> Reply-To: <us...@giraph.apache.org>
>>>> Date: Wednesday, 13 November 2013 11:15
>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>
>>>> Hi Rob,
>>>> Did you try with this command "mvn -Phadoop_yarn -Dhadoop.version=2.2.0
>>>> clean package -DskipTests"?
>>>> Regards,
>>>> Mohammad
>>>>
>>>>
>>>> On Wednesday, November 13, 2013 2:55 AM, Rob Vesse <
>>>> rvesse@dotnetrdf.org> wrote:
>>>> Hi all
>>>>
>>>> I'm having trouble getting Giraph to build and run against a Hadoop
>>>> 2.2.0 installation. I tried the hadoop_2.0.3 profile with no luck because
>>>> I get class cast errors when trying to run the examples.
>>>>
>>>> Is there a specific profile I should be building to support Hadoop
>>>> 2.2.0 or is this not supported yet?
>>>>
>>>> Thanks,
>>>>
>>>> Rob
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)
Posted by Alexandre Fonseca <al...@gmail.com>.
Hello Vishal,
Cross-posting from a different mailing list thread with the steps that allowed
me to compile and run SimpleShortestPaths on a pure-yarn cluster. Hope it
works for you too :)
I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of hadoop-2.2.0.
Here's what I did:
git clone https://git-wip-us.apache.org/repos/asf/giraph.git
cd giraph
git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile
With that I got valid jars which allowed me to run the SimpleShortestPaths
example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:
mkdir test_example
ln -s $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
$HADOOP_PREFIX/bin/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-
hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
giraph/input/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
hadoop-2.2.0-jar-with-dependencies.jar
###########
Notes:
* Wasn't able to do mvn package as it complained about not having the
dependencies giraph-hive, giraph-hbase, etc... I suspect this is just a matter
of changing the main pom.xml to include those modules with the hadoop 2.2.0
target or remove those dependencies from the giraph-dist if they no longer
apply.
* You may get some memory limit exceeded problems for the ApplicationMaster.
To fix, either increase the yarn.nodemanager.vmem-pmem-ratio parameter in yarn-
site.xml or apply my patch to prevent allocation of heap size equal to total
memory size, described at: https://issues.apache.org/jira/browse/GIRAPH-812
* In my case, the application didn't exit even after outputting the result and
having recognized that all nodes voted to terminate. I've opened an issue
in JIRA regarding this so if you suffer from the same, you can apply that patch
too: https://issues.apache.org/jira/browse/GIRAPH-811
###########
Relevant JIRA issues:
* https://issues.apache.org/jira/browse/GIRAPH-798
* https://issues.apache.org/jira/browse/GIRAPH-804
* https://issues.apache.org/jira/browse/GIRAPH-811
* https://issues.apache.org/jira/browse/GIRAPH-812
Best regards,
Alexandre
On Sunday 08 December 2013 14:52:15 Vishal Mishra wrote:
> Setup: hadoop 2.2.0 on a 2-node cluster on ubuntu 13.10.
> Giraph: sudo git clone https://github.com/apache/giraph.git
>
> Tried a few things in the past but was not able to get even one successful
> run for the basic example. Our cluster is healthy and we've been running MR
> and Hive successfully for over 3 weeks.
>
> Just started from scratch again and now I can't get past the build for the
> latest as several dependency checks are failing:
>
> [ERROR] Found a problem with the dependency
> commons-httpclient:commons-httpclient
> Resolved version is 3.0.1
> Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common
> [ERROR] Found a problem with the dependency commons-logging:commons-logging
> Resolved version is 1.0.4
> Version 1.1.1 was expected by artifact: org.apache.hadoop:hadoop-common
> [ERROR] Found a problem with the dependency io.netty:netty
> Resolved version is 3.5.3.Final
> Version 3.6.2.Final was expected by artifacts:
> org.apache.hadoop:hadoop-mapreduce-client-common,
> org.apache.hadoop:hadoop-mapreduce-client-core,
> org.apache.hadoop:hadoop-yarn-common,
> org.apache.hadoop:hadoop-yarn-server-common,
> org.apache.hadoop:hadoop-yarn-server-nodemanager,
> org.apache.hadoop:hadoop-yarn-server-resourcemanager,
> org.apache.hadoop:hadoop-yarn-server-tests:test-jar
> [ERROR] Found a problem with the dependency org.apache.zookeeper:zookeeper
> Resolved version is 3.3.3
> Version 3.4.5 was expected by artifacts: org.apache.hadoop:hadoop-common,
> org.apache.hadoop:hadoop-mapreduce-client-common,
> org.apache.hadoop:hadoop-yarn-server-common,
> org.apache.hadoop:hadoop-yarn-server-nodemanager,
> org.apache.hadoop:hadoop-yarn-server-resourcemanager,
> org.apache.hadoop:hadoop-yarn-server-tests:test-jar
> the building is failing due to dependency check issues
>
>
> Would really appreciate it if someone can share the correct branch and
> steps that have been successfully applied in a cluster. I'd be more than
> happy to document the steps and post them for everyone.
>
> Thanx,
> Vishal
>
> On Sun, Dec 8, 2013 at 10:23 AM, Eli Reisman
<ap...@gmail.com>wrote:
> > Hi all,
> >
> > It looks to me like you want to run the giraph-core jar at the beginning
> > of the "hadoop jar" call and supply the giraph-examples jar with -yj
> > argument.
> >
> > Also, you can place any -D type configuration settings you're sick of
> > supplying at the command line into giraph-site.xml just as you would with
> > Hadoop settings in one of its own site.xml files, and place these where
> > your local Hadoop client configs are and they should be picked up in your
> > application.
> >
> > Hope that helps,
> >
> > Eli
> >
> > On Thu, Nov 28, 2013 at 12:21 PM, Vishal Mishra
<mi...@gmail.com>wrote:
> >> I had a mistake in my command (moronic copy/paste zookeeper settings
> >> error).
> >> Updated and now hitting the following:
> >>
> >> ....
> >> 13/11/28 12:02:14 INFO yarn.GiraphYarnClient: Running Client
> >> 13/11/28 12:02:14 INFO client.RMProxy: Connecting to ResourceManager at
> >> master/192.168.1.100:8045
> >> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
> >> for, nodeId=kreator:46477, nodeAddresskreator:8042,
> >> nodeRackName/default-rack, nodeNumContainers7
> >> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
> >> for, nodeId=exotica:46645, nodeAddressexotica:8042,
> >> nodeRackName/default-rack, nodeNumContainers8
> >> Exception in thread "main" java.lang.RuntimeException: Giraph job
> >> requires 2 containers to run; cluster only hosts 15
> >>
> >> at
> >>
> >> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(Gi
> >> raphYarnClient.java:230) at
> >> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:125)
> >> ....
> >>
> >> It seems like some config is missing from our hadoop setup.
> >> Also, note that I'm working from the trunk branch.
> >>
> >> Thanx,
> >> Vishal
> >>
> >> On Wed, Nov 27, 2013 at 11:34 AM, Vishal Mishra
<mi...@gmail.com>wrote:
> >>> Hey Guys,
> >>> I'm trying giraph 1.0.0 on hadoop 2.2.0 (2-node cluster) but hitting the
> >>> same problem as outlined in this thread. Was there a resolution? If yes,
> >>> can you please share?
> >>> Completely new to giraph and hadoop so please excuse any omissions. Our
> >>> cluster is fully functional and running mapreduce and hive jobs cleanly.
> >>>
> >>> Command:
> >>> hadoop jar
> >>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-h
> >>> adoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner -D
> >>> giraph.zkList="zkNode.net:2081"
> >>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> >>> -vip /user/hduser/rrdata/tiny_graph.txt -vof
> >>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >>> /user/hduser/rrdata/output -w 1 -yj
> >>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-h
> >>> adoop-2.2.0-jar-with-dependencies.jar
> >>>
> >>> Trimmed Console Output:
> >>> 13/11/27 11:19:58 INFO yarn.GiraphYarnClient:
> >>> appattempt_1385518875931_0002_000001, State: ACCEPTED, Containers used:
> >>> 1
> >>> 13/11/27 11:20:00 ERROR yarn.GiraphYarnClient: Giraph:
> >>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
> >>> state, diagnostics show: Application application_1385518875931_0002
> >>> failed
> >>> 2 times due to AM Container for appattempt_1385518875931_0002_000002
> >>> exited
> >>>
> >>> with exitCode: 1 due to: Exception from container-launch:
> >>> org.apache.hadoop.util.Shell$ExitCodeException:
> >>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> >>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> >>>
> >>> at
> >>>
> >>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589
> >>> )
> >>> at
> >>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launc
> >>> hContainer(DefaultContainerExecutor.java:195)>>>
> >>> at
> >>>
> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont
> >>> ainerLaunch.call(ContainerLaunch.java:283) at
> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont
> >>> ainerLaunch.call(ContainerLaunch.java:79)>>>
> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>
> >>> at
> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> >>> a:1145)>>>
> >>> at
> >>>
> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> >>> va:615) at java.lang.Thread.run(Thread.java:744)
> >>>
> >>>
> >>> Thanx,
> >>> Vishal
> >>>
> >>> On Tue, Nov 19, 2013 at 3:16 PM, Mohammad Islam
<mi...@yahoo.com>wrote:
> >>>> Rob,
> >>>> I understand your frustration.
> >>>> I'm pre-occupied with some other stuffs in next couple of days.
> >>>> If no one replied by then, I will recreate the situation here and try
> >>>> to give you more concrete help.
> >>>>
> >>>> Regards,
> >>>> Mohammad
> >>>>
> >>>> On Tuesday, November 19, 2013 2:56 AM, Rob Vesse <
> >>>>
> >>>> rvesse@dotnetrdf.org> wrote:
> >>>> Mohammad
> >>>>
> >>>> Comments inline:
> >>>>
> >>>> From: Mohammad Islam <mi...@yahoo.com>
> >>>> Reply-To: <us...@giraph.apache.org>
> >>>> Date: Monday, 18 November 2013 22:24
> >>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
> >>>> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop
> >>>> 2.2.0)
> >>>>
> >>>> I used the following shell script to run PR where my current working
> >>>> directory contains the graph*.jar in Hadoop 2.1.1.
> >>>> . You can try something similar.
> >>>>
> >>>>
> >>>> I already am.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> HAD_VERSION=2.1.1-SNAPSHOT
> >>>> hadoop fs -rmr out
> >>>> export GIRAPH_HOME=/home/mislam/softwares/giraph
> >>>> #export
> >>>> GIRAPH_CORE=$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-h
> >>>> adoop-$HAD_VERSION-jar-with-dependencies.jar #export
> >>>> GIRAPH_EXAMPLE=$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.
> >>>> 0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
> >>>>
> >>>>
> >>>> CLASSPATH=$CLASSPATH:$GIRAPH_HOME/conf
> >>>>
> >>>>
> >>>> What is the point of putting the conf directory on the CLASSPATH? I
> >>>> assume this is only to pick up the giraph configuration but at the same
> >>>> time there is nothing to stop me putting my Giraph specific config in
> >>>> one
> >>>> of the other Hadoop configuration files e.g. core-site.xml?
> >>>>
> >>>> #CLASSPATH=$CLASSPATH:$GIRAPH_CORE
> >>>> #CLASSPATH=$CLASSPATH:$GIRAPH_EXAMPLE
> >>>>
> >>>> export CLASSPATH=$CLASSPATH
> >>>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
> >>>>
> >>>> echo $HADOOP_CLASSPATH
> >>>>
> >>>>
> >>>> I assume the point of the above is to ensure that the Giraph config
> >>>> file is on the class path Hadoop actually uses but adding this makes no
> >>>> difference to me, I get exactly the same failure as before. Modifying
> >>>> the
> >>>> logic to add the examples JAR onto the class path does not help either.
> >>>>
> >>>>
> >>>>
> >>>> hadoop --config $HADOOP_HOME/etc/hadoop jar
> >>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-depende
> >>>> ncies.jar>>>>
> >>>> org.apache.giraph.GiraphRunner
> >>>>
> >>>> org.apache.giraph.examples.SimplePageRankComputation -c
> >>>> org.apache.giraph.combiner.DoubleSumCombiner -w 1 -yh 1024 -yj
> >>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-depende
> >>>> ncies.jar -vif
> >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> >>>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -mc
> >>>> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMas
> >>>> terCompute -vip input -op out
> >>>>
> >>>>
> >>>> This is not the same command that I am running, also DoubleSumCombiner
> >>>> gives a ClassNotFoundException, you actually want
> >>>> DoubleSumMessageCombiner.
> >>>>
> >>>> If I adapt my script to run your command I experience the exact same
> >>>> error
> >>>>
> >>>> as I do trying to run the command I showed so I am still nowhere.
> >>>>
> >>>> I'm getting a little frustrated that no-one has any idea what the
> >>>> problem is here, either this is a simple misconfiguration on my part or
> >>>> a
> >>>> bug in Giraph for Hadoop 2.2.0 ? Given the thread so far and the fact
> >>>> that
> >>>> I can run MR based Giraph on Hadoop 1.2.1 this is looking increasingly
> >>>> unlikely I'm starting to lean heavily towards the latter option.
> >>>>
> >>>> Rob
> >>>>
> >>>> On Monday, November 18, 2013 2:24 AM, Rob Vesse
> >>>> <rv...@dotnetrdf.org>
> >>>>
> >>>> wrote:
> >>>> Hey all
> >>>>
> >>>> Bumping this thread because I am still completely blocked on this and
> >>>> hoping someone can help me. As explained earlier in the thread I have
> >>>> Giraph built for pure YARN on stable Hadoop 2.2.0 release like so:
> >>>>
> >>>> mvn –Phadoop_yarn –Dhadoop.version=2.2.0 –DskipTests=true clean install
> >>>>
> >>>> But when I try and run the simple shortest paths demo like so:
> >>>>
> >>>> $> bin/hadoop jar
> >>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-
> >>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> >>>> org.apache.giraph.GiraphRunner
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> >>>> -vip /user/rvesse/tiny_graph.txt -vof
> >>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >>>> /user/rvesse/shortestpaths -w 1 -yj
> >>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-
> >>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> >>>>
> >>>> The application fails and the YARN logs show the following error:
> >>>>
> >>>> Exception in thread "main" java.lang.NoClassDefFoundError:
> >>>> org/apache/giraph/yarn/GiraphApplicationMaster Caused by:
> >>>> java.lang.ClassNotFoundException:
> >>>> org.apache.giraph.yarn.GiraphApplicationMaster>>>>
> >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>>>
> >>>> Any ideas why the GiraphApplicationMaster would not be found? Have I
> >>>> just got something basic wrong in my configuration or is this a bug in
> >>>> Giraph?
> >>>>
> >>>>
> >>>> I found an earlier thread on the same topic
> >>>> (http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c5
> >>>> 1E5A4D1.40006@googlemail.com%3e) but it was never resolved hence why I
> >>>> have now changed the topic in the hope it will jog someone's memory
> >>>>
> >>>>
> >>>> Cheers,
> >>>>
> >>>>
> >>>> Rob
> >>>>
> >>>>
> >>>> From: Rob Vesse <rv...@dotnetrdf.org>
> >>>> Reply-To: <us...@giraph.apache.org>
> >>>> Date: Friday, 15 November 2013 14:14
> >>>> To: <us...@giraph.apache.org>
> >>>> Subject: Re: Giraph and Hadoop 2.2.0
> >>>>
> >>>> Adding the –yj argument to specify the YARN JARs doesn't make any
> >>>> difference.
> >>>>
> >>>> Here is the complete console output from my run:
> >>>>
> >>>> bin/hadoop jar
> >>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-
> >>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> >>>> org.apache.giraph.GiraphRunner
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> >>>> -vip /user/rvesse/tiny_graph.txt -vof
> >>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >>>> /user/rvesse/shortestpaths/3 -w 1 -yj
> >>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-
> >>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> >>>> 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info from
> >>>> SCDynamicStore
> >>>> 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load
> >>>> native-hadoop library for your platform... using builtin-java classes
> >>>> where
> >>>> applicable
> >>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format
> >>>> specified. Ensure your InputFormat does not require one.
> >>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output format
> >>>> specified. Ensure your OutputFormat does not require one.
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is:
> >>>> hdfs://localhost/user/rvesse/shortestpaths/3
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
> >>>> 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager at
> >>>> /0.0.0.0:8032
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from ASM
> >>>> for, nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042,
> >>>> nodeRackName/default-rack, nodeNumContainers0
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task heap
> >>>> size to 1024MB.
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application
> >>>> ID: application_1384523910015_0004
> >>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is
> >>>> deprecated. Instead, use mapreduce.job.id
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for
> >>>> the application master
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM
> >>>>
> >>>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share
> >>>> :/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP
> >>>> :_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib
> >>>> :/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoo
> >>>> :p/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPR
> >>>> :ED_HOME/share/hadoop/mapreduce/lib/*}>>>>
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap
> >>>> ....
> >>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir is
> >>>> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in
> >>>> LocalResources ::
> >>>> hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_13845239
> >>>> 10015_0004/giraph-conf.xml 13/11/15 14:11:25 INFO
> >>>> yarn.GiraphYarnClient: LIB JARS
> >>>>
> >>>> :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph
> >>>> :-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-
dependencies.jar>>>>
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /contrib/capacity-scheduler/*.jar
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /contrib/capacity-scheduler
> >>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient:
> >>>> ApplicationSumbissionContext for GiraphApplicationMaster launch
> >>>> container
> >>>> is populated.
> >>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application to
> >>>> ASM
> >>>> 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application
> >>>> application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
> >>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after
> >>>> submission :application_1384523910015_0004
> >>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
> >>>> container request was submitted to ResourceManager for job: Giraph:
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation
> >>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed:
> >>>> 0.82
> >>>> secs
> >>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
> >>>> appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers used:
> >>>> 1
> >>>> 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports
> >>>> FAILED
> >>>> state, diagnostics show: Application application_1384523910015_0004
> >>>> failed
> >>>> 2 times due to AM Container for appattempt_1384523910015_0004_000002
> >>>> exited
> >>>> with exitCode: 1 due to: Exception from container-launch:
> >>>> org.apache.hadoop.util.Shell$ExitCodeException:
> >>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> >>>>
> >>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> >>>>
> >>>> at
> >>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:58
> >>>> 9)
> >>>>
> >>>> at
> >>>>
> >>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.laun
> >>>> chContainer(DefaultContainerExecutor.java:195) at
> >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Con
> >>>> tainerLaunch.call(ContainerLaunch.java:283) at
> >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Con
> >>>> tainerLaunch.call(ContainerLaunch.java:79)>>>>
> >>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>>>
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>
> >>>> at
> >>>>
> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecut
> >>>> or.java:895) at
> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> >>>> ava:918)>>>>
> >>>> at java.lang.Thread.run(Thread.java:695)
> >>>>
> >>>> .Failing this attempt.. Failing the application.
> >>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS
> >>>> distributed cache directory for Giraph job.
> >>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED,
> >>>> total
> >>>> running time: 0 minutes, 3 seconds.
> >>>>
> >>>> And I see the same error in the logs:
> >>>>
> >>>> Exception in thread "main" java.lang.NoClassDefFoundError:
> >>>> org/apache/giraph/yarn/GiraphApplicationMaster Caused by:
> >>>> java.lang.ClassNotFoundException:
> >>>> org.apache.giraph.yarn.GiraphApplicationMaster>>>>
> >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>>>
> >>>> Any more ideas?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Rob
> >>>>
> >>>>
> >>>> From: Mohammad Islam <mi...@yahoo.com>
> >>>> Reply-To: <us...@giraph.apache.org>
> >>>> Date: Thursday, 14 November 2013 20:36
> >>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
> >>>> Subject: Re: Giraph and Hadoop 2.2.0
> >>>>
> >>>> I tried the following command few months back. I found you are not
> >>>> passing some params.
> >>>>
> >>>> bin/hadoop --config etc/hadoop jar
> >>>> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-exa
> >>>> mples-1.?.0-for-hadoop-???-jar-with-dependencies.jar
> >>>> org.apache.giraph.GiraphRunner
> >>>> org.apache.giraph.examples.SimplePageRankVertexOpt -c
> >>>> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj
> >>>> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif
> >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> >>>> -of
> >>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip
> >>>> /user/me/input -op /user/me/output_35w_20G_80m_1
> >>>>
> >>>> On Thursday, November 14, 2013 3:22 AM, Rob Vesse <
> >>>>
> >>>> rvesse@dotnetrdf.org> wrote:
> >>>> It's the same command line I showed earlier, just trying to run the
> >>>>
> >>>> example application:
> >>>>
> >>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
> >>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-
> >>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> >>>> org.apache.giraph.GiraphRunner
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> >>>> -vip /user/rvesse/tiny_graph.txt -vof
> >>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >>>> /user/rvesse/shortestpaths/2 -w 1
> >>>>
> >>>> Rob
> >>>>
> >>>> From: Mohammad Islam <mi...@yahoo.com>
> >>>> Reply-To: <us...@giraph.apache.org>
> >>>> Date: Thursday, 14 November 2013 05:57
> >>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
> >>>> Subject: Re: Giraph and Hadoop 2.2.0
> >>>>
> >>>> What are you running? PR?
> >>>> What is the command line?
> >>>>
> >>>> On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <
> >>>>
> >>>> rvesse@dotnetrdf.org> wrote:
> >>>> So I fixed the previous issue I was having which I reported and
> >>>>
> >>>> submitted a patch as GIRAPH-795
> >>>>
> >>>> I then went ahead with installing and configuring my external Zookeeper
> >>>> appropriately and tried running again but now the YARN application
> >>>> fails
> >>>> and I find the following error in the logs:
> >>>>
> >>>> Exception in thread "main" java.lang.NoClassDefFoundError:
> >>>> org/apache/giraph/yarn/GiraphApplicationMaster
> >>>>
> >>>> Caused by: java.lang.ClassNotFoundException:
> >>>> org.apache.giraph.yarn.GiraphApplicationMaster>>>>
> >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>>>
> >>>> Anybody have idea what the cause of this one might be or how to fix it?
> >>>>
> >>>>
> >>>> Cheers,
> >>>>
> >>>>
> >>>> Rob
> >>>>
> >>>>
> >>>> From: Rob Vesse <rv...@dotnetrdf.org>
> >>>> Reply-To: <us...@giraph.apache.org>
> >>>> Date: Wednesday, 13 November 2013 15:35
> >>>> To: <us...@giraph.apache.org>
> >>>> Subject: Re: Giraph and Hadoop 2.2.0
> >>>>
> >>>> Mohammad
> >>>>
> >>>> This got me a build that at least attempts to start to run but not I
> >>>> see the following:
> >>>>
> >>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
> >>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-
> >>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> >>>> org.apache.giraph.GiraphRunner
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> >>>> -vip /user/rvesse/tiny_graph.txt -vof
> >>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >>>> /user/rvesse/shortestpaths/2 -w 1
> >>>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info from
> >>>> SCDynamicStore
> >>>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load
> >>>> native-hadoop library for your platform... using builtin-java classes
> >>>> where
> >>>> applicable
> >>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format
> >>>> specified. Ensure your InputFormat does not require one.
> >>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output format
> >>>> specified. Ensure your OutputFormat does not require one.
> >>>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is:
> >>>> hdfs://localhost/user/rvesse/shortestpaths/2
> >>>> Exception in thread "main" java.lang.NullPointerException
> >>>>
> >>>> at
> >>>>
> >>>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported
> >>>> (GiraphYarnClient.java:460) at
> >>>> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
> >>>>
> >>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
> >>>>
> >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >>>>
> >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >>>>
> >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
> >>>>
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>
> >>>> at
> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> >>>> va:39)>>>>
> >>>> at
> >>>>
> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> >>>> rImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>
> >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> >>>>
> >>>> Which indirectly implies that I have some missing configuration to run
> >>>> Giraph on YARN. I don't think that should be a NPE and it looks to be
> >>>> a
> >>>> trivial fix so I'll try and fix and send a pull request for that.
> >>>>
> >>>> The error message I should receive if the NPE wasn't happening is the
> >>>> following:
> >>>>
> >>>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances:
> >>>> use a standalone ZooKeeper.
> >>>>
> >>>> What configuration options do I need to have set to point Giraph at an
> >>>> external Zookeeper?
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Rob
> >>>>
> >>>> From: Mohammad Islam <mi...@yahoo.com>
> >>>> Reply-To: <us...@giraph.apache.org>
> >>>> Date: Wednesday, 13 November 2013 11:15
> >>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
> >>>> Subject: Re: Giraph and Hadoop 2.2.0
> >>>>
> >>>> Hi Rob,
> >>>> Did you try with this command "mvn -Phadoop_yarn -Dhadoop.version=2.2.0
> >>>> clean package -DskipTests"?
> >>>> Regards,
> >>>> Mohammad
> >>>>
> >>>> On Wednesday, November 13, 2013 2:55 AM, Rob Vesse <
> >>>>
> >>>> rvesse@dotnetrdf.org> wrote:
> >>>> Hi all
> >>>>
> >>>> I'm having trouble getting Giraph to build and run against a Hadoop
> >>>> 2.2.0 installation. I tried the hadoop_2.0.3 profile with no luck
> >>>> because
> >>>> I get class cast errors when trying to run the examples.
> >>>>
> >>>> Is there a specific profile I should be building to support Hadoop
> >>>> 2.2.0 or is this not supported yet?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Rob
Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)
Posted by Vishal Mishra <mi...@gmail.com>.
Setup: hadoop 2.2.0 on a 2-node cluster on ubuntu 13.10.
Giraph: sudo git clone https://github.com/apache/giraph.git
Tried a few things in the past but was not able to get even one successful
run for the basic example. Our cluster is healthy and we've been running MR
and Hive successfully for over 3 weeks.
Just started from scratch again and now I can't get past the build for the
latest as several dependency checks are failing:
[ERROR] Found a problem with the dependency
commons-httpclient:commons-httpclient
Resolved version is 3.0.1
Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common
[ERROR] Found a problem with the dependency commons-logging:commons-logging
Resolved version is 1.0.4
Version 1.1.1 was expected by artifact: org.apache.hadoop:hadoop-common
[ERROR] Found a problem with the dependency io.netty:netty
Resolved version is 3.5.3.Final
Version 3.6.2.Final was expected by artifacts:
org.apache.hadoop:hadoop-mapreduce-client-common,
org.apache.hadoop:hadoop-mapreduce-client-core,
org.apache.hadoop:hadoop-yarn-common,
org.apache.hadoop:hadoop-yarn-server-common,
org.apache.hadoop:hadoop-yarn-server-nodemanager,
org.apache.hadoop:hadoop-yarn-server-resourcemanager,
org.apache.hadoop:hadoop-yarn-server-tests:test-jar
[ERROR] Found a problem with the dependency org.apache.zookeeper:zookeeper
Resolved version is 3.3.3
Version 3.4.5 was expected by artifacts: org.apache.hadoop:hadoop-common,
org.apache.hadoop:hadoop-mapreduce-client-common,
org.apache.hadoop:hadoop-yarn-server-common,
org.apache.hadoop:hadoop-yarn-server-nodemanager,
org.apache.hadoop:hadoop-yarn-server-resourcemanager,
org.apache.hadoop:hadoop-yarn-server-tests:test-jar
the building is failing due to dependency check issues
Would really appreciate it if someone can share the correct branch and
steps that have been successfully applied in a cluster. I'd be more than
happy to document the steps and post them for everyone.
Thanx,
Vishal
On Sun, Dec 8, 2013 at 10:23 AM, Eli Reisman <ap...@gmail.com>wrote:
> Hi all,
>
> It looks to me like you want to run the giraph-core jar at the beginning
> of the "hadoop jar" call and supply the giraph-examples jar with -yj
> argument.
>
> Also, you can place any -D type configuration settings you're sick of
> supplying at the command line into giraph-site.xml just as you would with
> Hadoop settings in one of its own site.xml files, and place these where
> your local Hadoop client configs are and they should be picked up in your
> application.
>
> Hope that helps,
>
> Eli
>
>
>
> On Thu, Nov 28, 2013 at 12:21 PM, Vishal Mishra <mi...@gmail.com>wrote:
>
>> I had a mistake in my command (moronic copy/paste zookeeper settings
>> error).
>> Updated and now hitting the following:
>>
>> ....
>> 13/11/28 12:02:14 INFO yarn.GiraphYarnClient: Running Client
>> 13/11/28 12:02:14 INFO client.RMProxy: Connecting to ResourceManager at
>> master/192.168.1.100:8045
>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
>> for, nodeId=kreator:46477, nodeAddresskreator:8042,
>> nodeRackName/default-rack, nodeNumContainers7
>> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from ASM
>> for, nodeId=exotica:46645, nodeAddressexotica:8042,
>> nodeRackName/default-rack, nodeNumContainers8
>> Exception in thread "main" java.lang.RuntimeException: Giraph job
>> requires 2 containers to run; cluster only hosts 15
>> at
>> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230)
>> at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:125)
>> ....
>>
>> It seems like some config is missing from our hadoop setup.
>> Also, note that I'm working from the trunk branch.
>>
>> Thanx,
>> Vishal
>>
>>
>> On Wed, Nov 27, 2013 at 11:34 AM, Vishal Mishra <mi...@gmail.com>wrote:
>>
>>> Hey Guys,
>>> I'm trying giraph 1.0.0 on hadoop 2.2.0 (2-node cluster) but hitting the
>>> same problem as outlined in this thread. Was there a resolution? If yes,
>>> can you please share?
>>> Completely new to giraph and hadoop so please excuse any omissions. Our
>>> cluster is fully functional and running mapreduce and hive jobs cleanly.
>>>
>>> Command:
>>> hadoop jar
>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>> org.apache.giraph.GiraphRunner -D giraph.zkList="zkNode.net:2081"
>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>> -vip /user/hduser/rrdata/tiny_graph.txt -vof
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>> /user/hduser/rrdata/output -w 1 -yj
>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>
>>> Trimmed Console Output:
>>> 13/11/27 11:19:58 INFO yarn.GiraphYarnClient:
>>> appattempt_1385518875931_0002_000001, State: ACCEPTED, Containers used: 1
>>> 13/11/27 11:20:00 ERROR yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>> state, diagnostics show: Application application_1385518875931_0002 failed
>>> 2 times due to AM Container for appattempt_1385518875931_0002_000002 exited
>>> with exitCode: 1 due to: Exception from container-launch:
>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>> at
>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>>
>>>
>>> Thanx,
>>> Vishal
>>>
>>>
>>> On Tue, Nov 19, 2013 at 3:16 PM, Mohammad Islam <mi...@yahoo.com>wrote:
>>>
>>>> Rob,
>>>> I understand your frustration.
>>>> I'm pre-occupied with some other stuffs in next couple of days.
>>>> If no one replied by then, I will recreate the situation here and try
>>>> to give you more concrete help.
>>>>
>>>> Regards,
>>>> Mohammad
>>>>
>>>>
>>>> On Tuesday, November 19, 2013 2:56 AM, Rob Vesse <
>>>> rvesse@dotnetrdf.org> wrote:
>>>> Mohammad
>>>>
>>>> Comments inline:
>>>>
>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>> Reply-To: <us...@giraph.apache.org>
>>>> Date: Monday, 18 November 2013 22:24
>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop
>>>> 2.2.0)
>>>>
>>>> I used the following shell script to run PR where my current working
>>>> directory contains the graph*.jar in Hadoop 2.1.1.
>>>> . You can try something similar.
>>>>
>>>>
>>>> I already am.
>>>>
>>>>
>>>>
>>>>
>>>> HAD_VERSION=2.1.1-SNAPSHOT
>>>> hadoop fs -rmr out
>>>> export GIRAPH_HOME=/home/mislam/softwares/giraph
>>>> #export
>>>> GIRAPH_CORE=$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>> #export
>>>> GIRAPH_EXAMPLE=$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>>
>>>>
>>>> CLASSPATH=$CLASSPATH:$GIRAPH_HOME/conf
>>>>
>>>>
>>>> What is the point of putting the conf directory on the CLASSPATH? I
>>>> assume this is only to pick up the giraph configuration but at the same
>>>> time there is nothing to stop me putting my Giraph specific config in one
>>>> of the other Hadoop configuration files e.g. core-site.xml?
>>>>
>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_CORE
>>>> #CLASSPATH=$CLASSPATH:$GIRAPH_EXAMPLE
>>>>
>>>> export CLASSPATH=$CLASSPATH
>>>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
>>>>
>>>> echo $HADOOP_CLASSPATH
>>>>
>>>>
>>>> I assume the point of the above is to ensure that the Giraph config
>>>> file is on the class path Hadoop actually uses but adding this makes no
>>>> difference to me, I get exactly the same failure as before. Modifying the
>>>> logic to add the examples JAR onto the class path does not help either.
>>>>
>>>>
>>>>
>>>> hadoop --config $HADOOP_HOME/etc/hadoop jar
>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimplePageRankComputation -c
>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 1 -yh 1024 -yj
>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
>>>> -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -mc
>>>> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
>>>> -vip input -op out
>>>>
>>>>
>>>> This is not the same command that I am running, also DoubleSumCombiner
>>>> gives a ClassNotFoundException, you actually want DoubleSumMessageCombiner.
>>>> If I adapt my script to run your command I experience the exact same error
>>>> as I do trying to run the command I showed so I am still nowhere.
>>>>
>>>> I'm getting a little frustrated that no-one has any idea what the
>>>> problem is here, either this is a simple misconfiguration on my part or a
>>>> bug in Giraph for Hadoop 2.2.0 ? Given the thread so far and the fact that
>>>> I can run MR based Giraph on Hadoop 1.2.1 this is looking increasingly
>>>> unlikely I'm starting to lean heavily towards the latter option.
>>>>
>>>> Rob
>>>>
>>>>
>>>>
>>>> On Monday, November 18, 2013 2:24 AM, Rob Vesse <rv...@dotnetrdf.org>
>>>> wrote:
>>>> Hey all
>>>>
>>>> Bumping this thread because I am still completely blocked on this and
>>>> hoping someone can help me. As explained earlier in the thread I have
>>>> Giraph built for pure YARN on stable Hadoop 2.2.0 release like so:
>>>>
>>>> mvn –Phadoop_yarn –Dhadoop.version=2.2.0 –DskipTests=true clean install
>>>>
>>>> But when I try and run the simple shortest paths demo like so:
>>>>
>>>> $> bin/hadoop jar
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/rvesse/shortestpaths -w 1 -yj
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>>
>>>> The application fails and the YARN logs show the following error:
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>
>>>>
>>>>
>>>>
>>>> Any ideas why the GiraphApplicationMaster would not be found? Have I just got something basic wrong in my configuration or is this a bug in Giraph?
>>>>
>>>>
>>>> I found an earlier thread on the same topic (http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c51E5A4D1.40006@googlemail.com%3e) but it was never resolved hence why I have now changed the topic in the hope it will jog someone's memory
>>>>
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> Rob
>>>>
>>>>
>>>> From: Rob Vesse <rv...@dotnetrdf.org>
>>>> Reply-To: <us...@giraph.apache.org>
>>>> Date: Friday, 15 November 2013 14:14
>>>> To: <us...@giraph.apache.org>
>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>
>>>> Adding the –yj argument to specify the YARN JARs doesn't make any
>>>> difference.
>>>>
>>>> Here is the complete console output from my run:
>>>>
>>>> bin/hadoop jar
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/rvesse/shortestpaths/3 -w 1 -yj
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info from
>>>> SCDynamicStore
>>>> 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load
>>>> native-hadoop library for your platform... using builtin-java classes where
>>>> applicable
>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format
>>>> specified. Ensure your InputFormat does not require one.
>>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output format
>>>> specified. Ensure your OutputFormat does not require one.
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is:
>>>> hdfs://localhost/user/rvesse/shortestpaths/3
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
>>>> 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager at
>>>> /0.0.0.0:8032
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from ASM
>>>> for, nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042,
>>>> nodeRackName/default-rack, nodeNumContainers0
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task heap
>>>> size to 1024MB.
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application
>>>> ID: application_1384523910015_0004
>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is
>>>> deprecated. Instead, use mapreduce.job.id
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for
>>>> the application master
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM
>>>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir is
>>>> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in
>>>> LocalResources ::
>>>> hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_1384523910015_0004/giraph-conf.xml
>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS
>>>> :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
>>>> /contrib/capacity-scheduler/*.jar
>>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
>>>> /contrib/capacity-scheduler
>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient:
>>>> ApplicationSumbissionContext for GiraphApplicationMaster launch container
>>>> is populated.
>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application to
>>>> ASM
>>>> 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application
>>>> application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032
>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after
>>>> submission :application_1384523910015_0004
>>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>>>> container request was submitted to ResourceManager for job: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation
>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.82
>>>> secs
>>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
>>>> appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers used: 1
>>>> 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>>> state, diagnostics show: Application application_1384523910015_0004 failed
>>>> 2 times due to AM Container for appattempt_1384523910015_0004_000002 exited
>>>> with exitCode: 1 due to: Exception from container-launch:
>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>> at
>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>> at java.lang.Thread.run(Thread.java:695)
>>>>
>>>>
>>>> .Failing this attempt.. Failing the application.
>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS
>>>> distributed cache directory for Giraph job.
>>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
>>>> running time: 0 minutes, 3 seconds.
>>>>
>>>> And I see the same error in the logs:
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster
>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>
>>>>
>>>> Any more ideas?
>>>>
>>>>
>>>>
>>>>
>>>> Rob
>>>>
>>>>
>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>> Reply-To: <us...@giraph.apache.org>
>>>> Date: Thursday, 14 November 2013 20:36
>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>
>>>> I tried the following command few months back. I found you are not
>>>> passing some params.
>>>>
>>>> bin/hadoop --config etc/hadoop jar
>>>> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-examples-1.?.0-for-hadoop-???-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimplePageRankVertexOpt -c
>>>> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj
>>>> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -of
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip
>>>> /user/me/input -op /user/me/output_35w_20G_80m_1
>>>>
>>>>
>>>> On Thursday, November 14, 2013 3:22 AM, Rob Vesse <
>>>> rvesse@dotnetrdf.org> wrote:
>>>> It's the same command line I showed earlier, just trying to run the
>>>> example application:
>>>>
>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/rvesse/shortestpaths/2 -w 1
>>>>
>>>> Rob
>>>>
>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>> Reply-To: <us...@giraph.apache.org>
>>>> Date: Thursday, 14 November 2013 05:57
>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>
>>>> What are you running? PR?
>>>> What is the command line?
>>>>
>>>>
>>>>
>>>> On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <
>>>> rvesse@dotnetrdf.org> wrote:
>>>> So I fixed the previous issue I was having which I reported and
>>>> submitted a patch as GIRAPH-795
>>>>
>>>> I then went ahead with installing and configuring my external Zookeeper
>>>> appropriately and tried running again but now the YARN application fails
>>>> and I find the following error in the logs:
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> org/apache/giraph/yarn/GiraphApplicationMaster
>>>> Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>
>>>>
>>>> Anybody have idea what the cause of this one might be or how to fix it?
>>>>
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> Rob
>>>>
>>>>
>>>> From: Rob Vesse <rv...@dotnetrdf.org>
>>>> Reply-To: <us...@giraph.apache.org>
>>>> Date: Wednesday, 13 November 2013 15:35
>>>> To: <us...@giraph.apache.org>
>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>
>>>> Mohammad
>>>>
>>>> This got me a build that at least attempts to start to run but not I
>>>> see the following:
>>>>
>>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
>>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/rvesse/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/rvesse/shortestpaths/2 -w 1
>>>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info from
>>>> SCDynamicStore
>>>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load
>>>> native-hadoop library for your platform... using builtin-java classes where
>>>> applicable
>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format
>>>> specified. Ensure your InputFormat does not require one.
>>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output format
>>>> specified. Ensure your OutputFormat does not require one.
>>>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is:
>>>> hdfs://localhost/user/rvesse/shortestpaths/2
>>>> Exception in thread "main" java.lang.NullPointerException
>>>> at
>>>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(GiraphYarnClient.java:460)
>>>> at
>>>> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116)
>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>
>>>> Which indirectly implies that I have some missing configuration to run
>>>> Giraph on YARN. I don't think that should be a NPE and it looks to be a
>>>> trivial fix so I'll try and fix and send a pull request for that.
>>>>
>>>> The error message I should receive if the NPE wasn't happening is the
>>>> following:
>>>>
>>>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances:
>>>> use a standalone ZooKeeper.
>>>>
>>>> What configuration options do I need to have set to point Giraph at an
>>>> external Zookeeper?
>>>>
>>>> Cheers,
>>>>
>>>> Rob
>>>>
>>>> From: Mohammad Islam <mi...@yahoo.com>
>>>> Reply-To: <us...@giraph.apache.org>
>>>> Date: Wednesday, 13 November 2013 11:15
>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>> Subject: Re: Giraph and Hadoop 2.2.0
>>>>
>>>> Hi Rob,
>>>> Did you try with this command "mvn -Phadoop_yarn -Dhadoop.version=2.2.0
>>>> clean package -DskipTests"?
>>>> Regards,
>>>> Mohammad
>>>>
>>>>
>>>> On Wednesday, November 13, 2013 2:55 AM, Rob Vesse <
>>>> rvesse@dotnetrdf.org> wrote:
>>>> Hi all
>>>>
>>>> I'm having trouble getting Giraph to build and run against a Hadoop
>>>> 2.2.0 installation. I tried the hadoop_2.0.3 profile with no luck because
>>>> I get class cast errors when trying to run the examples.
>>>>
>>>> Is there a specific profile I should be building to support Hadoop
>>>> 2.2.0 or is this not supported yet?
>>>>
>>>> Thanks,
>>>>
>>>> Rob
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>