You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Kristen Hardwick <kh...@spryinc.com> on 2014/01/08 21:53:29 UTC
Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster
Hi all,
When running the SimplePageRankComputation Giraph example, I get the
following error in the YARN logs (and nothing else):
Could not find or load main class
org.apache.giraph.yarn.GiraphApplicationMaster
On the command line, I see:
14/01/08 14:40:03 INFO utils.ConfigurationUtils: No edge input format
specified. Ensure your InputFormat does not require one.
14/01/08 14:40:03 INFO utils.ConfigurationUtils: No edge output format
specified. Ensure your OutputFormat does not require one.
14/01/08 14:40:04 INFO yarn.GiraphYarnClient: Final output path is:
hdfs://hadoop2.j7.master:8020/user/spry/output/shortestpaths
14/01/08 14:40:04 INFO yarn.GiraphYarnClient: Running Client
14/01/08 14:40:04 INFO client.RMProxy: Connecting to ResourceManager at
hadoop2.j7.master/127.0.0.1:8050
14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Got node report from ASM for,
nodeId=localhost:45454, nodeAddresslocalhost:8042,
nodeRackName/default-rack, nodeNumContainers0
14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Defaulting per-task heap size
to 1024MB.
14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Obtained new Application ID:
application_1389197443854_0014
14/01/08 14:40:05 INFO Configuration.deprecation: mapred.job.id is
deprecated. Instead, use mapreduce.job.id
14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Set the environment for the
application master
14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Environment for AM
:{CLASSPATH=${CLASSPATH}:./*:/etc/hadoop/conf:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-mapreduce/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
14/01/08 14:40:05 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
14/01/08 14:40:05 INFO Configuration.deprecation: mapred.output.dir is
deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/01/08 14:40:07 INFO yarn.YarnUtils: Registered file in LocalResources ::
hdfs://hadoop2.j7.master:8020/user/spry/giraph_yarn_jar_cache/application_1389197443854_0014/giraph-conf.xml
14/01/08 14:40:07 INFO yarn.GiraphYarnClient: LIB JARS
:giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar,giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
14/01/08 14:40:07 INFO yarn.YarnUtils: Class path name .
14/01/08 14:40:07 INFO yarn.YarnUtils: base path checking .
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name null
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking null
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name /etc/hadoop/conf
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking /etc/hadoop/conf
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop/lib/*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop/lib
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop/.//*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop/./
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-hdfs/./
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-hdfs/./
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-hdfs/lib/*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-hdfs/lib
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-hdfs/.//*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-hdfs/./
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-yarn/lib/*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-yarn/lib
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-yarn/.//*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-yarn/./
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-mapreduce/lib/*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-mapreduce/lib
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-mapreduce/.//*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-mapreduce/./
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/share/java/mysql-connector-java-5.1.17.jar
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/share/java/mysql-connector-java-5.1.17.jar
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/share/java/mysql-connector-java.jar
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/share/java/mysql-connector-java.jar
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-mapreduce/*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-mapreduce
14/01/08 14:40:08 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext
for GiraphApplicationMaster launch container is populated.
14/01/08 14:40:08 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/01/08 14:40:09 INFO impl.YarnClientImpl: Submitted application
application_1389197443854_0014 to ResourceManager at hadoop2.j7.master/
127.0.0.1:8050
14/01/08 14:40:09 INFO yarn.GiraphYarnClient: Got new appId after
submission :application_1389197443854_0014
14/01/08 14:40:09 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
container request was submitted to ResourceManager for job: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation
14/01/08 14:40:09 INFO yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.96
secs
14/01/08 14:40:09 INFO yarn.GiraphYarnClient:
appattempt_1389197443854_0014_000001, State: ACCEPTED, Containers used: 1
14/01/08 14:40:13 ERROR yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
state, diagnostics show: Application application_1389197443854_0014 failed
2 times due to AM Container for appattempt_1389197443854_0014_000002 exited
with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
.Failing this attempt.. Failing the application.
14/01/08 14:40:13 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed
cache directory for Giraph job.
14/01/08 14:40:13 INFO yarn.GiraphYarnClient: Completed Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
running time: 0 minutes, 4 seconds.
Googling led me to this mailing list and some suggestions to use the -yj
flag to include the giraph-core and giraph-example jar files, but that did
not help. I was unable to find any posts that said they were able to
resolve the issue (and included what they ended up trying). Does anyone see
anything wrong with using this command:
hadoop jar
/usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner -Dgiraph.zkList="localhost:2181"
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/spry/input/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/spry/output/shortestpaths -w 1 -yj
giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar,giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
For background:
I compiled Giraph with this command, and everything built successfully
except "Apache Giraph Distribution" which it doesn't seem like I need:
mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests clean package
I am running with the following components:
Single node cluster
Giraph 1.1
Hadoop 2.2.0 (Hortonworks)
Java 1.7.0_45
Any help is appreciated. Thanks in advance!
Kristen Hardwick
Re: Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster
Posted by Eli Reisman <ap...@gmail.com>.
Hi,
Well it gets as far as receiving a container and appId from the RM and it
looks like for just a sec the AM gets up and running then dies. The command
line looks good (Mohammed who did the update to Hadoop 2.2 should probably
confirm this) but the one thing I notice is that you're calling one giraph
jar in the "hadoop jar" call but including another copy of the same jar on
your source repo path with -yj. This won't matter if both are identical
builds but otherwise this could be trouble? Just a guess.
Anyway...hopefully you have it figured out. Post more stack traces if not.
Thanks!
On Wed, Jan 8, 2014 at 12:53 PM, Kristen Hardwick <kh...@spryinc.com>wrote:
> Hi all,
>
> When running the SimplePageRankComputation Giraph example, I get the
> following error in the YARN logs (and nothing else):
>
> Could not find or load main class
> org.apache.giraph.yarn.GiraphApplicationMaster
>
> On the command line, I see:
>
> 14/01/08 14:40:03 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
> 14/01/08 14:40:03 INFO utils.ConfigurationUtils: No edge output format
> specified. Ensure your OutputFormat does not require one.
> 14/01/08 14:40:04 INFO yarn.GiraphYarnClient: Final output path is:
> hdfs://hadoop2.j7.master:8020/user/spry/output/shortestpaths
> 14/01/08 14:40:04 INFO yarn.GiraphYarnClient: Running Client
> 14/01/08 14:40:04 INFO client.RMProxy: Connecting to ResourceManager at
> hadoop2.j7.master/127.0.0.1:8050
> 14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Got node report from ASM
> for, nodeId=localhost:45454, nodeAddresslocalhost:8042,
> nodeRackName/default-rack, nodeNumContainers0
> 14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Defaulting per-task heap
> size to 1024MB.
> 14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Obtained new Application ID:
> application_1389197443854_0014
> 14/01/08 14:40:05 INFO Configuration.deprecation: mapred.job.id is
> deprecated. Instead, use mapreduce.job.id
> 14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Set the environment for the
> application master
> 14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Environment for AM
> :{CLASSPATH=${CLASSPATH}:./*:/etc/hadoop/conf:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-mapreduce/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
> 14/01/08 14:40:05 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
> 14/01/08 14:40:05 INFO Configuration.deprecation: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 14/01/08 14:40:07 INFO yarn.YarnUtils: Registered file in LocalResources
> ::
> hdfs://hadoop2.j7.master:8020/user/spry/giraph_yarn_jar_cache/application_1389197443854_0014/giraph-conf.xml
> 14/01/08 14:40:07 INFO yarn.GiraphYarnClient: LIB JARS
> :giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar,giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> 14/01/08 14:40:07 INFO yarn.YarnUtils: Class path name .
> 14/01/08 14:40:07 INFO yarn.YarnUtils: base path checking .
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name null
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking null
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name /etc/hadoop/conf
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking /etc/hadoop/conf
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
> /usr/lib/hadoop/lib/*
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
> /usr/lib/hadoop/lib
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop/.//*
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
> /usr/lib/hadoop/./
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
> /usr/lib/hadoop-hdfs/./
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
> /usr/lib/hadoop-hdfs/./
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
> /usr/lib/hadoop-hdfs/lib/*
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
> /usr/lib/hadoop-hdfs/lib
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
> /usr/lib/hadoop-hdfs/.//*
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
> /usr/lib/hadoop-hdfs/./
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
> /usr/lib/hadoop-yarn/lib/*
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
> /usr/lib/hadoop-yarn/lib
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
> /usr/lib/hadoop-yarn/.//*
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
> /usr/lib/hadoop-yarn/./
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
> /usr/lib/hadoop-mapreduce/lib/*
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
> /usr/lib/hadoop-mapreduce/lib
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
> /usr/lib/hadoop-mapreduce/.//*
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
> /usr/lib/hadoop-mapreduce/./
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
> /usr/share/java/mysql-connector-java-5.1.17.jar
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
> /usr/share/java/mysql-connector-java-5.1.17.jar
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
> /usr/share/java/mysql-connector-java.jar
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
> /usr/share/java/mysql-connector-java.jar
> 14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
> /usr/lib/hadoop-mapreduce/*
> 14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
> /usr/lib/hadoop-mapreduce
> 14/01/08 14:40:08 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext
> for GiraphApplicationMaster launch container is populated.
> 14/01/08 14:40:08 INFO yarn.GiraphYarnClient: Submitting application to ASM
> 14/01/08 14:40:09 INFO impl.YarnClientImpl: Submitted application
> application_1389197443854_0014 to ResourceManager at hadoop2.j7.master/
> 127.0.0.1:8050
> 14/01/08 14:40:09 INFO yarn.GiraphYarnClient: Got new appId after
> submission :application_1389197443854_0014
> 14/01/08 14:40:09 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
> container request was submitted to ResourceManager for job: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation
> 14/01/08 14:40:09 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.96
> secs
> 14/01/08 14:40:09 INFO yarn.GiraphYarnClient:
> appattempt_1389197443854_0014_000001, State: ACCEPTED, Containers used: 1
> 14/01/08 14:40:13 ERROR yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
> state, diagnostics show: Application application_1389197443854_0014 failed
> 2 times due to AM Container for appattempt_1389197443854_0014_000002 exited
> with exitCode: 1 due to: Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
>
> .Failing this attempt.. Failing the application.
> 14/01/08 14:40:13 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed
> cache directory for Giraph job.
> 14/01/08 14:40:13 INFO yarn.GiraphYarnClient: Completed Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
> running time: 0 minutes, 4 seconds.
>
> Googling led me to this mailing list and some suggestions to use the -yj
> flag to include the giraph-core and giraph-example jar files, but that did
> not help. I was unable to find any posts that said they were able to
> resolve the issue (and included what they ended up trying). Does anyone see
> anything wrong with using this command:
>
> hadoop jar
> /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner -Dgiraph.zkList="localhost:2181"
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /user/spry/input/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/spry/output/shortestpaths -w 1 -yj
> giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar,giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>
> For background:
>
> I compiled Giraph with this command, and everything built successfully
> except "Apache Giraph Distribution" which it doesn't seem like I need:
> mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests clean package
>
> I am running with the following components:
> Single node cluster
> Giraph 1.1
> Hadoop 2.2.0 (Hortonworks)
> Java 1.7.0_45
>
> Any help is appreciated. Thanks in advance!
> Kristen Hardwick
>