java.lang.ClassNotFoundException for VertexOutputFormat

[Marcin Biczak <ma...@gmail.com>]

Hi

I have very strange problem regarding ClassNotFoundException for vertex
output format (my own implementation). When I run my program I get an
exception:

java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException:
org.test.giraph.utils.writers.generic.SamplingOutputFormat
        at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
        at
org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:134)
        at
org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56)
        at org.apache.hadoop.mapred.Task.initialize(Task.java:490)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:352)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:253)

I have came across this exception earlier and I've solved it with the use
of "-libjars" parameter in which I pass the path to
giraph-0.2-SNAPSHOT-jar-with-dependencies.jar and myJob.jar (which I
execute). This solution has worked very nice for me, I used it in a number
of programs (with my own InputFormat and OutputFormat), till today. I've
created a small program (TestVertex) to test one solution and I used a
NewOutputFormat created for the test program. When I received
ClassNotFoundException I thought I have made a mistake somewhere, so I've
checked. I couldn't find any bugs, so I replaced the NewOutputFormat with a
one used across other programs (OldOutputFormat). I got the same exception.
for the old one as well. I have cleaned and build the whole myJob.jar and I
was still getting the exception (both for New and Old). So I tested the
original program (same myJob.jar) in which I'm using the OldOutputFormat
and the program works. Since then I tried couple of OutputFormat and none
of them worked (ClassNotFoundException). Then I commented the whole "body"
of TestVertex and I've paste some old working code (with OldOutputFormat).
The only difference between TestVertex and old working code is the class
name. Final I received the ClassNotFoundException again (again the original
code works, I've tested it). It looks like only the TestVertex class has
this problem, no matter what OutputFormat I will use I am always getting
the exception (all other programs which use my OutputFormats work
correctly).

Another interesting aspect is the fact that if I use Giraph provided
OutputFormat (for example JsonBase64VertexOutputFormat) I get a
ClassNotFoundException but for the TestVertex. I've checked the myJob.jar
and all mentioned classes are present in the jar in correct places.

I run out of ideas any suggestions will be greatly appreciated.

Some conf info
I am using an older Giraph revision (1340869) and Hadoop 0.20.203 in
pseudo-distributed mode.
I have created the HADOOP_CLASSPATH which holds path to
giraph-0.2-SNAPSHOT-jar-with-dependencies.jar.
I execute job in the following way: hadoop jar myJob.jar
org.test.giraph.Test -libjars
/path/to/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar,/path/to/myJob.jar
/path/to/input /path/to/output <job params>

regards
marcin

Re: java.lang.ClassNotFoundException for VertexOutputFormat

[Marcin Biczak <ma...@gmail.com>]

Hi again

I have found a solution to my "problem". I simply added myJobs.jar to the
HADOOP_CLASSPATH. Also "-libjars" parameter is not needed any more. But it
still puzzles me why the previous "solution" "suddenly" stopped working (I
haven't update the giraph revision etc).

HADOOP_CLASSPATH contains girpah.jar and myJobs.jar

regards
marcin

2012/11/8 Marcin Biczak <ma...@gmail.com>

> Hi
>
> I have very strange problem regarding ClassNotFoundException for vertex
> output format (my own implementation). When I run my program I get an
> exception:
>
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException:
> org.test.giraph.utils.writers.generic.SamplingOutputFormat
>         at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
>         at
> org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:134)
>         at
> org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56)
>         at org.apache.hadoop.mapred.Task.initialize(Task.java:490)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:352)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>         at org.apache.hadoop.mapred.Child.main(Child.java:253)
>
> I have came across this exception earlier and I've solved it with the use
> of "-libjars" parameter in which I pass the path to
> giraph-0.2-SNAPSHOT-jar-with-dependencies.jar and myJob.jar (which I
> execute). This solution has worked very nice for me, I used it in a number
> of programs (with my own InputFormat and OutputFormat), till today. I've
> created a small program (TestVertex) to test one solution and I used a
> NewOutputFormat created for the test program. When I received
> ClassNotFoundException I thought I have made a mistake somewhere, so I've
> checked. I couldn't find any bugs, so I replaced the NewOutputFormat with a
> one used across other programs (OldOutputFormat). I got the same exception.
> for the old one as well. I have cleaned and build the whole myJob.jar and I
> was still getting the exception (both for New and Old). So I tested the
> original program (same myJob.jar) in which I'm using the OldOutputFormat
> and the program works. Since then I tried couple of OutputFormat and none
> of them worked (ClassNotFoundException). Then I commented the whole "body"
> of TestVertex and I've paste some old working code (with OldOutputFormat).
> The only difference between TestVertex and old working code is the class
> name. Final I received the ClassNotFoundException again (again the original
> code works, I've tested it). It looks like only the TestVertex class has
> this problem, no matter what OutputFormat I will use I am always getting
> the exception (all other programs which use my OutputFormats work
> correctly).
>
> Another interesting aspect is the fact that if I use Giraph provided
> OutputFormat (for example JsonBase64VertexOutputFormat) I get a
> ClassNotFoundException but for the TestVertex. I've checked the myJob.jar
> and all mentioned classes are present in the jar in correct places.
>
> I run out of ideas any suggestions will be greatly appreciated.
>
> Some conf info
> I am using an older Giraph revision (1340869) and Hadoop 0.20.203 in
> pseudo-distributed mode.
> I have created the HADOOP_CLASSPATH which holds path to
> giraph-0.2-SNAPSHOT-jar-with-dependencies.jar.
> I execute job in the following way: hadoop jar myJob.jar
> org.test.giraph.Test -libjars
> /path/to/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar,/path/to/myJob.jar
> /path/to/input /path/to/output <job params>
>
> regards
> marcin
>



-- 
martin