You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@crunch.apache.org by Aditi Viswanathan <Ad...@247incc.onmicrosoft.com> on 2015/01/19 09:19:02 UTC

IncompatibleClassChangeError while running the crunch-examples job jar

Hi,

I'm trying to run the crunch-examples job jar file with this command:

hadoop jar crunch-examples-*-job.jar org.apache.crunch.examples.WordCount input/input.txt output

I'm getting the following error:

Exception in thread "Thread-5" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
        at org.apache.crunch.impl.mr.run.CrunchInputFormat.getSplits(CrunchInputFormat.java:44)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
        at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.submit(CrunchControlledJob.java:340)
        at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startReadyJobs(CrunchJobControl.java:277)
        at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:316)
        at org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:113)
        at org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:55)
        at org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:84)
        at java.lang.Thread.run(Thread.java:744)
Exception in thread "main" java.lang.NullPointerException
        at org.apache.crunch.examples.WordCount.run(WordCount.java:72)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.crunch.examples.WordCount.main(WordCount.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)


My Hadoop Version is:
Hadoop 2.3.0-cdh5.0.0

I've also added the jars to my Hadoop classpath like so:

-bash-3.2$ hadoop classpath
/etc/hadoop/conf:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop/.//*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/./:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-yarn/.//*:/var/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/var/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/var/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.64/lib/hadoop/lib/*:/home/samikr/crunch-jars/lib/hadoop-core-1.2.1.jar:/home/samikr/wc.jar:/usr/java/jdk1.7.0_51/lib/tools.jar:/etc/hadoop/conf:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop/.//*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/./:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-yarn/.//*:/var/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/var/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/var/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.64/lib/hadoop/lib/*:/home/samikr/crunch-examples-0.11.0-job.jar:/home/samikr/crunch-jars/lib/crunch-hbase-0.11.0.jar:/home/samikr/crunch-jars/lib/crunch-core-0.11.0.jar:/home/samikr/crunch-jars/lib/guava-11.0.2.jar::

(this includes the crunch-examples jar as well as the crunch-core and hadoop-core jar)


I have no idea how to solve this error - as far as I can tell, hadoop-core-1.2.1.jar has org/apache/hadoop/mapred/JobClient.class, so there shouldnt be a problem finding this class.
Thanks in advance for any help

RE: IncompatibleClassChangeError while running the crunch-examples job jar

Posted by Aditi Viswanathan <Ad...@247-inc.com>.
It worked when I rebuilt with mvn clean install -Dcrunch.platform=2
I also had a faulty internet connection from my VM which I didn’t detect initially because the maven build was successful anyway. So I first fixed that, and then activated the Hadoop-2 profile after which the program worked like a charm ☺

Thank you!

From: Micah Whitacre [mailto:mkwhitacre@gmail.com]
Sent: Monday, January 19, 2015 6:59 PM
To: user@crunch.apache.org
Subject: Re: IncompatibleClassChangeError while running the crunch-examples job jar

Aditi,
   what version of the jar are you running with?  Did a deployed version or did you build it locally?  If you built locally try activating the "hadoop-2" profile.  This can be done with either of the following commands:

mvn clean install -P hadoop-2
or
mvn clean install -Dcrunch.platform=2

The current version of master targets the Hadoop2 code stream.

If you are running with the latest released version are you running with the 0.11.0-hadoop2 version of the examples jar?

On Mon, Jan 19, 2015 at 2:19 AM, Aditi Viswanathan <Ad...@247incc.onmicrosoft.com>> wrote:
Hi,

I'm trying to run the crunch-examples job jar file with this command:

hadoop jar crunch-examples-*-job.jar org.apache.crunch.examples.WordCount input/input.txt output

I'm getting the following error:

Exception in thread "Thread-5" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
        at org.apache.crunch.impl.mr.run.CrunchInputFormat.getSplits(CrunchInputFormat.java:44)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
        at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.submit(CrunchControlledJob.java:340)
        at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startReadyJobs(CrunchJobControl.java:277)
        at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:316)
        at org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:113)
        at org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:55)
        at org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:84)
        at java.lang.Thread.run(Thread.java:744)
Exception in thread "main" java.lang.NullPointerException
        at org.apache.crunch.examples.WordCount.run(WordCount.java:72)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.crunch.examples.WordCount.main(WordCount.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)


My Hadoop Version is:
Hadoop 2.3.0-cdh5.0.0

I’ve also added the jars to my Hadoop classpath like so:

-bash-3.2$ hadoop classpath
/etc/hadoop/conf:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop/.//*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/./:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-yarn/.//*:/var/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/var/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/var/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.64/lib/hadoop/lib/*:/home/samikr/crunch-jars/lib/hadoop-core-1.2.1.jar:/home/samikr/wc.jar:/usr/java/jdk1.7.0_51/lib/tools.jar:/etc/hadoop/conf:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop/.//*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/./:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-yarn/.//*:/var/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/var/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/var/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.64/lib/hadoop/lib/*:/home/samikr/crunch-examples-0.11.0-job.jar:/home/samikr/crunch-jars/lib/crunch-hbase-0.11.0.jar:/home/samikr/crunch-jars/lib/crunch-core-0.11.0.jar:/home/samikr/crunch-jars/lib/guava-11.0.2.jar::

(this includes the crunch-examples jar as well as the crunch-core and hadoop-core jar)


I have no idea how to solve this error - as far as I can tell, hadoop-core-1.2.1.jar has org/apache/hadoop/mapred/JobClient.class, so there shouldnt be a problem finding this class.
Thanks in advance for any help

Re: IncompatibleClassChangeError while running the crunch-examples job jar

Posted by Micah Whitacre <mk...@gmail.com>.
Aditi,
   what version of the jar are you running with?  Did a deployed version or
did you build it locally?  If you built locally try activating the
"hadoop-2" profile.  This can be done with either of the following commands:

mvn clean install -P hadoop-2
or
mvn clean install -Dcrunch.platform=2

The current version of master targets the Hadoop2 code stream.

If you are running with the latest released version are you running with
the 0.11.0-hadoop2 version of the examples jar?

On Mon, Jan 19, 2015 at 2:19 AM, Aditi Viswanathan <
Aditi.V@247incc.onmicrosoft.com> wrote:

>  Hi,
>
>
>
> I'm trying to run the crunch-examples job jar file with this command:
>
>
>
> hadoop jar crunch-examples-*-job.jar org.apache.crunch.examples.WordCount
> input/input.txt output
>
>
>
> I'm getting the following error:
>
>
>
> Exception in thread "Thread-5" java.lang.IncompatibleClassChangeError:
> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
> expected
>
>         at
> org.apache.crunch.impl.mr.run.CrunchInputFormat.getSplits(CrunchInputFormat.java:44)
>
>         at
> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)
>
>         at
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)
>
>         at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
>
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
>
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
>
>         at
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.submit(CrunchControlledJob.java:340)
>
>         at
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startReadyJobs(CrunchJobControl.java:277)
>
>         at
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:316)
>
>         at
> org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:113)
>
>         at
> org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:55)
>
>         at
> org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:84)
>
>         at java.lang.Thread.run(Thread.java:744)
>
> Exception in thread "main" java.lang.NullPointerException
>
>         at org.apache.crunch.examples.WordCount.run(WordCount.java:72)
>
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
>         at org.apache.crunch.examples.WordCount.main(WordCount.java:76)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>         at java.lang.reflect.Method.invoke(Method.java:606)
>
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
>
>
>
>
> My Hadoop Version is:
>
> Hadoop 2.3.0-cdh5.0.0
>
>
>
> I’ve also added the jars to my Hadoop classpath like so:
>
>
>
> -bash-3.2$ hadoop classpath
>
>
> /etc/hadoop/conf:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop/.//*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/./:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-yarn/.//*:/var/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/var/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/var/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.64/lib/hadoop/lib/*:/home/samikr/crunch-jars/lib/hadoop-core-1.2.1.jar:/home/samikr/wc.jar:/usr/java/jdk1.7.0_51/lib/tools.jar:/etc/hadoop/conf:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop/.//*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/./:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/var/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.48/lib/hadoop/libexec/../../hadoop-yarn/.//*:/var/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/var/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/var/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.64/lib/hadoop/lib/*:/home/samikr/crunch-examples-0.11.0-job.jar:/home/samikr/crunch-jars/lib/crunch-hbase-0.11.0.jar:/home/samikr/crunch-jars/lib/crunch-core-0.11.0.jar:/home/samikr/crunch-jars/lib/guava-11.0.2.jar::
>
>
>
> (this includes the crunch-examples jar as well as the crunch-core and
> hadoop-core jar)
>
>
>
>
>
> I have no idea how to solve this error - as far as I can tell,
> hadoop-core-1.2.1.jar has org/apache/hadoop/mapred/JobClient.class, so
> there shouldnt be a problem finding this class.
>
> Thanks in advance for any help
>
>
>