[SOLVED] Re: Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster

[Phillip Rhodes <mo...@gmail.com>]

And finally, success is at hand!  This is a bit quirky, but here's what
fixed it:

My command line originally looked like this:

$> hadoop jar /home/prhodes/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/prhodes/input/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/prhodes/giraph_output/shortestpaths -w 4 -yj
/home/prhodes/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-dependencies.jar

I vaguely recalled seeing a mention of another user creating a symlink
to the Giraph jar files, and originally I thought he did that as a convenience.
But some intuition suggested that the fully qualified paths might actually
be causing a problem, so I did

$> ln -s /home/prhodes/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-dependencies.jar

to create a link to that jar in the current directory, then re-ran the
command as:

$> hadoop jar giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/prhodes/input/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/prhodes/giraph_output/shortestpaths -w 4 -yj
giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-dependencies.jar

and viola! It works...


Cheers,


Phil

On Wed, Mar 11, 2015 at 11:37 PM, Phillip Rhodes
<mo...@gmail.com> wrote:
> Gang:
>
> I am getting further with my attempt to get Giraph running on a YARN cluster,
> but now I'm stuck at this error:
>
> Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster
>
> I've tried everything I can find in previous messages on this topic,
> to no avail. My command line is like this:
>
> hadoop jar /home/prhodes/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /user/prhodes/input/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/prhodes/giraph_output/shortestpaths -w 4 -yj
> /home/prhodes/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-dependencies.jar,/home/prhodes/giraph/giraph-core/target/giraph-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-dependencies.jar
>
> and those jars are in those locations on each node in the cluster.
> But yet it still complains about not finding the
> GiraphApplicationMaster.   Also, I've verified that the jar file does
> contain the GiraphApplicationMaster.class.
>
> Any ideas what else might be causing the problem, or any workarounds?
> I thought about distributing the Giraph jars to each node and
> physically putting them on the Hadoop Classpath, but the Maven build
> builds fat jars by default, which would probably cause issues doing
> that.  Any other suggestions or ideas?
>
>
> Phil
> ---
> This message optimized for indexing by NSA PRISM