You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jonathan Poon <jk...@ucdavis.edu> on 2014/02/22 02:11:30 UTC
Having trouble adding external JAR to MapReduce Program
Hi Everyone,
I'm running into trouble adding the Avro JAR into my MapReduce program. I
do the following to try to add the Avro JAR:
export
HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
export
LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
/user/jonathanpoon/avro /user/jonathanpoon/output
However, I get the following error:
14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
attempt_201402191318_0014_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.avro.mapreduce.AvroKeyInputFormat
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
at
org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:187)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.mapreduce.AvroKeyInputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
... 8 more
Am I placing the Avro JAR files in the improper place?
Thanks for your help!
Jonathan
Re: Having trouble adding external JAR to MapReduce Program
Posted by Jonathan Poon <jk...@ucdavis.edu>.
Thanks for the suggestions! I will give them a look and try to see it it
will work!
Jonathan
On Fri, Feb 21, 2014 at 6:35 PM, Azuryy Yu <az...@gmail.com> wrote:
> Hi,
>
> you cannot add jar like this way.
>
> please look at DistributeCache in the Hadoop Java Doc.
>
> please call DistributeCache.addArchive() in your main Class before submit
> the MR job.
>
>
> On Sat, Feb 22, 2014 at 9:30 AM, Gaurav Gupta <ga...@datatorrent.com>wrote:
>
>> Jonathan,
>>
>>
>>
>> You have to make sure that the jar is available on the nodes where the
>> map reduce job is running. Setting the HADOOP_CLASSPATH on the single node
>> doesn't work.
>>
>> You can use -libjars to the hadoop command line.
>>
>>
>>
>> Thanks
>>
>> Gaurav
>>
>>
>>
>> *From:* Jonathan Poon [mailto:jkpoon@ucdavis.edu]
>> *Sent:* Friday, February 21, 2014 5:12 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Having trouble adding external JAR to MapReduce Program
>>
>>
>>
>> Hi Everyone,
>>
>> I'm running into trouble adding the Avro JAR into my MapReduce program.
>> I do the following to try to add the Avro JAR:
>>
>> export
>> HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
>>
>> export
>> LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
>>
>> hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
>> /user/jonathanpoon/avro /user/jonathanpoon/output
>>
>> However, I get the following error:
>>
>> 14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
>> attempt_201402191318_0014_m_000001_2, Status : FAILED
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.avro.mapreduce.AvroKeyInputFormat
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
>> at
>> org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:187)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.avro.mapreduce.AvroKeyInputFormat
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:270)
>> at
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
>> ... 8 more
>>
>>
>> Am I placing the Avro JAR files in the improper place?
>>
>> Thanks for your help!
>>
>> Jonathan
>>
>>
>>
>
>
Re: Having trouble adding external JAR to MapReduce Program
Posted by Jonathan Poon <jk...@ucdavis.edu>.
Thanks for the suggestions! I will give them a look and try to see it it
will work!
Jonathan
On Fri, Feb 21, 2014 at 6:35 PM, Azuryy Yu <az...@gmail.com> wrote:
> Hi,
>
> you cannot add jar like this way.
>
> please look at DistributeCache in the Hadoop Java Doc.
>
> please call DistributeCache.addArchive() in your main Class before submit
> the MR job.
>
>
> On Sat, Feb 22, 2014 at 9:30 AM, Gaurav Gupta <ga...@datatorrent.com>wrote:
>
>> Jonathan,
>>
>>
>>
>> You have to make sure that the jar is available on the nodes where the
>> map reduce job is running. Setting the HADOOP_CLASSPATH on the single node
>> doesn't work.
>>
>> You can use -libjars to the hadoop command line.
>>
>>
>>
>> Thanks
>>
>> Gaurav
>>
>>
>>
>> *From:* Jonathan Poon [mailto:jkpoon@ucdavis.edu]
>> *Sent:* Friday, February 21, 2014 5:12 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Having trouble adding external JAR to MapReduce Program
>>
>>
>>
>> Hi Everyone,
>>
>> I'm running into trouble adding the Avro JAR into my MapReduce program.
>> I do the following to try to add the Avro JAR:
>>
>> export
>> HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
>>
>> export
>> LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
>>
>> hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
>> /user/jonathanpoon/avro /user/jonathanpoon/output
>>
>> However, I get the following error:
>>
>> 14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
>> attempt_201402191318_0014_m_000001_2, Status : FAILED
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.avro.mapreduce.AvroKeyInputFormat
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
>> at
>> org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:187)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.avro.mapreduce.AvroKeyInputFormat
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:270)
>> at
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
>> ... 8 more
>>
>>
>> Am I placing the Avro JAR files in the improper place?
>>
>> Thanks for your help!
>>
>> Jonathan
>>
>>
>>
>
>
Re: Having trouble adding external JAR to MapReduce Program
Posted by Jonathan Poon <jk...@ucdavis.edu>.
Thanks for the suggestions! I will give them a look and try to see it it
will work!
Jonathan
On Fri, Feb 21, 2014 at 6:35 PM, Azuryy Yu <az...@gmail.com> wrote:
> Hi,
>
> you cannot add jar like this way.
>
> please look at DistributeCache in the Hadoop Java Doc.
>
> please call DistributeCache.addArchive() in your main Class before submit
> the MR job.
>
>
> On Sat, Feb 22, 2014 at 9:30 AM, Gaurav Gupta <ga...@datatorrent.com>wrote:
>
>> Jonathan,
>>
>>
>>
>> You have to make sure that the jar is available on the nodes where the
>> map reduce job is running. Setting the HADOOP_CLASSPATH on the single node
>> doesn't work.
>>
>> You can use -libjars to the hadoop command line.
>>
>>
>>
>> Thanks
>>
>> Gaurav
>>
>>
>>
>> *From:* Jonathan Poon [mailto:jkpoon@ucdavis.edu]
>> *Sent:* Friday, February 21, 2014 5:12 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Having trouble adding external JAR to MapReduce Program
>>
>>
>>
>> Hi Everyone,
>>
>> I'm running into trouble adding the Avro JAR into my MapReduce program.
>> I do the following to try to add the Avro JAR:
>>
>> export
>> HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
>>
>> export
>> LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
>>
>> hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
>> /user/jonathanpoon/avro /user/jonathanpoon/output
>>
>> However, I get the following error:
>>
>> 14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
>> attempt_201402191318_0014_m_000001_2, Status : FAILED
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.avro.mapreduce.AvroKeyInputFormat
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
>> at
>> org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:187)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.avro.mapreduce.AvroKeyInputFormat
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:270)
>> at
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
>> ... 8 more
>>
>>
>> Am I placing the Avro JAR files in the improper place?
>>
>> Thanks for your help!
>>
>> Jonathan
>>
>>
>>
>
>
Re: Having trouble adding external JAR to MapReduce Program
Posted by Jonathan Poon <jk...@ucdavis.edu>.
Thanks for the suggestions! I will give them a look and try to see it it
will work!
Jonathan
On Fri, Feb 21, 2014 at 6:35 PM, Azuryy Yu <az...@gmail.com> wrote:
> Hi,
>
> you cannot add jar like this way.
>
> please look at DistributeCache in the Hadoop Java Doc.
>
> please call DistributeCache.addArchive() in your main Class before submit
> the MR job.
>
>
> On Sat, Feb 22, 2014 at 9:30 AM, Gaurav Gupta <ga...@datatorrent.com>wrote:
>
>> Jonathan,
>>
>>
>>
>> You have to make sure that the jar is available on the nodes where the
>> map reduce job is running. Setting the HADOOP_CLASSPATH on the single node
>> doesn't work.
>>
>> You can use -libjars to the hadoop command line.
>>
>>
>>
>> Thanks
>>
>> Gaurav
>>
>>
>>
>> *From:* Jonathan Poon [mailto:jkpoon@ucdavis.edu]
>> *Sent:* Friday, February 21, 2014 5:12 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Having trouble adding external JAR to MapReduce Program
>>
>>
>>
>> Hi Everyone,
>>
>> I'm running into trouble adding the Avro JAR into my MapReduce program.
>> I do the following to try to add the Avro JAR:
>>
>> export
>> HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
>>
>> export
>> LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
>>
>> hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
>> /user/jonathanpoon/avro /user/jonathanpoon/output
>>
>> However, I get the following error:
>>
>> 14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
>> attempt_201402191318_0014_m_000001_2, Status : FAILED
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.avro.mapreduce.AvroKeyInputFormat
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
>> at
>> org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:187)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.avro.mapreduce.AvroKeyInputFormat
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:270)
>> at
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
>> ... 8 more
>>
>>
>> Am I placing the Avro JAR files in the improper place?
>>
>> Thanks for your help!
>>
>> Jonathan
>>
>>
>>
>
>
Re: Having trouble adding external JAR to MapReduce Program
Posted by Azuryy Yu <az...@gmail.com>.
Hi,
you cannot add jar like this way.
please look at DistributeCache in the Hadoop Java Doc.
please call DistributeCache.addArchive() in your main Class before submit
the MR job.
On Sat, Feb 22, 2014 at 9:30 AM, Gaurav Gupta <ga...@datatorrent.com>wrote:
> Jonathan,
>
>
>
> You have to make sure that the jar is available on the nodes where the map
> reduce job is running. Setting the HADOOP_CLASSPATH on the single node
> doesn't work.
>
> You can use -libjars to the hadoop command line.
>
>
>
> Thanks
>
> Gaurav
>
>
>
> *From:* Jonathan Poon [mailto:jkpoon@ucdavis.edu]
> *Sent:* Friday, February 21, 2014 5:12 PM
> *To:* user@hadoop.apache.org
> *Subject:* Having trouble adding external JAR to MapReduce Program
>
>
>
> Hi Everyone,
>
> I'm running into trouble adding the Avro JAR into my MapReduce program. I
> do the following to try to add the Avro JAR:
>
> export
> HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
>
> export
> LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
>
> hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
> /user/jonathanpoon/avro /user/jonathanpoon/output
>
> However, I get the following error:
>
> 14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
> attempt_201402191318_0014_m_000001_2, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.avro.mapreduce.AvroKeyInputFormat
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
> at
> org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:187)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.avro.mapreduce.AvroKeyInputFormat
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
> ... 8 more
>
>
> Am I placing the Avro JAR files in the improper place?
>
> Thanks for your help!
>
> Jonathan
>
>
>
Re: Having trouble adding external JAR to MapReduce Program
Posted by Azuryy Yu <az...@gmail.com>.
Hi,
you cannot add jar like this way.
please look at DistributeCache in the Hadoop Java Doc.
please call DistributeCache.addArchive() in your main Class before submit
the MR job.
On Sat, Feb 22, 2014 at 9:30 AM, Gaurav Gupta <ga...@datatorrent.com>wrote:
> Jonathan,
>
>
>
> You have to make sure that the jar is available on the nodes where the map
> reduce job is running. Setting the HADOOP_CLASSPATH on the single node
> doesn't work.
>
> You can use -libjars to the hadoop command line.
>
>
>
> Thanks
>
> Gaurav
>
>
>
> *From:* Jonathan Poon [mailto:jkpoon@ucdavis.edu]
> *Sent:* Friday, February 21, 2014 5:12 PM
> *To:* user@hadoop.apache.org
> *Subject:* Having trouble adding external JAR to MapReduce Program
>
>
>
> Hi Everyone,
>
> I'm running into trouble adding the Avro JAR into my MapReduce program. I
> do the following to try to add the Avro JAR:
>
> export
> HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
>
> export
> LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
>
> hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
> /user/jonathanpoon/avro /user/jonathanpoon/output
>
> However, I get the following error:
>
> 14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
> attempt_201402191318_0014_m_000001_2, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.avro.mapreduce.AvroKeyInputFormat
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
> at
> org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:187)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.avro.mapreduce.AvroKeyInputFormat
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
> ... 8 more
>
>
> Am I placing the Avro JAR files in the improper place?
>
> Thanks for your help!
>
> Jonathan
>
>
>
Re: Having trouble adding external JAR to MapReduce Program
Posted by Azuryy Yu <az...@gmail.com>.
Hi,
you cannot add jar like this way.
please look at DistributeCache in the Hadoop Java Doc.
please call DistributeCache.addArchive() in your main Class before submit
the MR job.
On Sat, Feb 22, 2014 at 9:30 AM, Gaurav Gupta <ga...@datatorrent.com>wrote:
> Jonathan,
>
>
>
> You have to make sure that the jar is available on the nodes where the map
> reduce job is running. Setting the HADOOP_CLASSPATH on the single node
> doesn't work.
>
> You can use -libjars to the hadoop command line.
>
>
>
> Thanks
>
> Gaurav
>
>
>
> *From:* Jonathan Poon [mailto:jkpoon@ucdavis.edu]
> *Sent:* Friday, February 21, 2014 5:12 PM
> *To:* user@hadoop.apache.org
> *Subject:* Having trouble adding external JAR to MapReduce Program
>
>
>
> Hi Everyone,
>
> I'm running into trouble adding the Avro JAR into my MapReduce program. I
> do the following to try to add the Avro JAR:
>
> export
> HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
>
> export
> LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
>
> hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
> /user/jonathanpoon/avro /user/jonathanpoon/output
>
> However, I get the following error:
>
> 14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
> attempt_201402191318_0014_m_000001_2, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.avro.mapreduce.AvroKeyInputFormat
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
> at
> org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:187)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.avro.mapreduce.AvroKeyInputFormat
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
> ... 8 more
>
>
> Am I placing the Avro JAR files in the improper place?
>
> Thanks for your help!
>
> Jonathan
>
>
>
Re: Having trouble adding external JAR to MapReduce Program
Posted by Azuryy Yu <az...@gmail.com>.
Hi,
you cannot add jar like this way.
please look at DistributeCache in the Hadoop Java Doc.
please call DistributeCache.addArchive() in your main Class before submit
the MR job.
On Sat, Feb 22, 2014 at 9:30 AM, Gaurav Gupta <ga...@datatorrent.com>wrote:
> Jonathan,
>
>
>
> You have to make sure that the jar is available on the nodes where the map
> reduce job is running. Setting the HADOOP_CLASSPATH on the single node
> doesn't work.
>
> You can use -libjars to the hadoop command line.
>
>
>
> Thanks
>
> Gaurav
>
>
>
> *From:* Jonathan Poon [mailto:jkpoon@ucdavis.edu]
> *Sent:* Friday, February 21, 2014 5:12 PM
> *To:* user@hadoop.apache.org
> *Subject:* Having trouble adding external JAR to MapReduce Program
>
>
>
> Hi Everyone,
>
> I'm running into trouble adding the Avro JAR into my MapReduce program. I
> do the following to try to add the Avro JAR:
>
> export
> HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
>
> export
> LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
>
> hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
> /user/jonathanpoon/avro /user/jonathanpoon/output
>
> However, I get the following error:
>
> 14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
> attempt_201402191318_0014_m_000001_2, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.avro.mapreduce.AvroKeyInputFormat
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
> at
> org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:187)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.avro.mapreduce.AvroKeyInputFormat
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
> ... 8 more
>
>
> Am I placing the Avro JAR files in the improper place?
>
> Thanks for your help!
>
> Jonathan
>
>
>
RE: Having trouble adding external JAR to MapReduce Program
Posted by Gaurav Gupta <ga...@datatorrent.com>.
Jonathan,
You have to make sure that the jar is available on the nodes where the map
reduce job is running. Setting the HADOOP_CLASSPATH on the single node
doesn't work.
You can use -libjars to the hadoop command line.
Thanks
Gaurav
From: Jonathan Poon [mailto:jkpoon@ucdavis.edu]
Sent: Friday, February 21, 2014 5:12 PM
To: user@hadoop.apache.org
Subject: Having trouble adding external JAR to MapReduce Program
Hi Everyone,
I'm running into trouble adding the Avro JAR into my MapReduce program. I
do the following to try to add the Avro JAR:
export
HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avr
o-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro
-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
export
LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/a
vro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/av
ro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
/user/jonathanpoon/avro /user/jonathanpoon/output
However, I get the following error:
14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
attempt_201402191318_0014_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.avro.mapreduce.AvroKeyInputFormat
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
at
org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:1
87)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.mapreduce.AvroKeyInputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
... 8 more
Am I placing the Avro JAR files in the improper place?
Thanks for your help!
Jonathan
RE: Having trouble adding external JAR to MapReduce Program
Posted by Gaurav Gupta <ga...@datatorrent.com>.
Jonathan,
You have to make sure that the jar is available on the nodes where the map
reduce job is running. Setting the HADOOP_CLASSPATH on the single node
doesn't work.
You can use -libjars to the hadoop command line.
Thanks
Gaurav
From: Jonathan Poon [mailto:jkpoon@ucdavis.edu]
Sent: Friday, February 21, 2014 5:12 PM
To: user@hadoop.apache.org
Subject: Having trouble adding external JAR to MapReduce Program
Hi Everyone,
I'm running into trouble adding the Avro JAR into my MapReduce program. I
do the following to try to add the Avro JAR:
export
HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avr
o-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro
-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
export
LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/a
vro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/av
ro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
/user/jonathanpoon/avro /user/jonathanpoon/output
However, I get the following error:
14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
attempt_201402191318_0014_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.avro.mapreduce.AvroKeyInputFormat
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
at
org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:1
87)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.mapreduce.AvroKeyInputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
... 8 more
Am I placing the Avro JAR files in the improper place?
Thanks for your help!
Jonathan
RE: Having trouble adding external JAR to MapReduce Program
Posted by Gaurav Gupta <ga...@datatorrent.com>.
Jonathan,
You have to make sure that the jar is available on the nodes where the map
reduce job is running. Setting the HADOOP_CLASSPATH on the single node
doesn't work.
You can use -libjars to the hadoop command line.
Thanks
Gaurav
From: Jonathan Poon [mailto:jkpoon@ucdavis.edu]
Sent: Friday, February 21, 2014 5:12 PM
To: user@hadoop.apache.org
Subject: Having trouble adding external JAR to MapReduce Program
Hi Everyone,
I'm running into trouble adding the Avro JAR into my MapReduce program. I
do the following to try to add the Avro JAR:
export
HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avr
o-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro
-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
export
LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/a
vro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/av
ro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
/user/jonathanpoon/avro /user/jonathanpoon/output
However, I get the following error:
14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
attempt_201402191318_0014_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.avro.mapreduce.AvroKeyInputFormat
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
at
org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:1
87)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.mapreduce.AvroKeyInputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
... 8 more
Am I placing the Avro JAR files in the improper place?
Thanks for your help!
Jonathan
RE: Having trouble adding external JAR to MapReduce Program
Posted by Gaurav Gupta <ga...@datatorrent.com>.
Jonathan,
You have to make sure that the jar is available on the nodes where the map
reduce job is running. Setting the HADOOP_CLASSPATH on the single node
doesn't work.
You can use -libjars to the hadoop command line.
Thanks
Gaurav
From: Jonathan Poon [mailto:jkpoon@ucdavis.edu]
Sent: Friday, February 21, 2014 5:12 PM
To: user@hadoop.apache.org
Subject: Having trouble adding external JAR to MapReduce Program
Hi Everyone,
I'm running into trouble adding the Avro JAR into my MapReduce program. I
do the following to try to add the Avro JAR:
export
HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avr
o-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro
-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"
export
LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/a
vro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/av
ro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"
hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
/user/jonathanpoon/avro /user/jonathanpoon/output
However, I get the following error:
14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
attempt_201402191318_0014_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.avro.mapreduce.AvroKeyInputFormat
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
at
org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:1
87)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.mapreduce.AvroKeyInputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
... 8 more
Am I placing the Avro JAR files in the improper place?
Thanks for your help!
Jonathan