You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "C.V.Krishnakumar Iyer" <f2...@gmail.com> on 2011/01/11 19:10:40 UTC
libjars options
Hi,
Could anyone please guide me as to how to use the -libjars option in HDFS?
I have added the necessary jar file (the hbase jar - to be precise) to the classpath of the node where I am starting the job.
The following is the format that i am invoking:
bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated by Commas)> <Arguments to our main class>
bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a
Despite this, I find that I get the java.lang.ClassNotFoundException error! :(
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
The strange thing is that there is another MR job I have that runs perfectly with the libjars option! Could anybody tell me what I am doing wrong? One more thing - not sure if it is relevant : I am using the new Hadoop MapReduce API.
Thanks in advance!
Regards,
Krishnakumar.
Re: libjars options
Posted by "C.V.Krishnakumar Iyer" <f2...@gmail.com>.
Hi,
Thanks a lot Alex! using GenericOptionsParser solved the issue. Previously I had used Tool and had assumed that it would take care of this.
Regards,
Krishna.
On Jan 11, 2011, at 12:48 PM, Alex Kozlov wrote:
> There is also a blog that I recently wrote, if it helps
> http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job
>
> On Tue, Jan 11, 2011 at 12:33 PM, Alex Kozlov <al...@cloudera.com> wrote:
>
>> Have you implemented GenericOptionsParser? Do you see your jar in the *
>> mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view
>> via a JT Web UI)?
>>
>> --
>> Alex Kozlov
>> Solutions Architect
>> Cloudera, Inc
>> twitter: alexvk2009
>> <http://www.cloudera.com/company/press-center/hadoop-world-nyc/>
>>
>>
>> On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer <
>> f2004403@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have tried that as well, using -files <jar file> But it still gives the
>>> exact same error. Any other thing that I could try?
>>>
>>> Thanks,
>>> Krishna.
>>>
>>> On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:
>>>
>>>> Refer to Alex Kozlov's answer on 12/11/10
>>>>
>>>> On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
>>>> <f2...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Could anyone please guide me as to how to use the -libjars option in
>>> HDFS?
>>>>>
>>>>> I have added the necessary jar file (the hbase jar - to be precise) to
>>> the
>>>>> classpath of the node where I am starting the job.
>>>>>
>>>>> The following is the format that i am invoking:
>>>>> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars
>>> (separated by
>>>>> Commas)> <Arguments to our main class>
>>>>>
>>>>> bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars
>>>>> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a
>>>>>
>>>>> Despite this, I find that I get the java.lang.ClassNotFoundException
>>>>> error! :(
>>>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>>>> java.lang.ClassNotFoundException:
>>>>> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>>>>> at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
>>>>> at
>>>>>
>>> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
>>>>> at
>>>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
>>>>> at
>>>>> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
>>>>> at
>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>>> Caused by: java.lang.RuntimeException:
>>> java.lang.ClassNotFoundException:
>>>>> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>>>>> at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
>>>>> at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
>>>>>
>>>>> The strange thing is that there is another MR job I have that runs
>>>>> perfectly with the libjars option! Could anybody tell me what I am
>>> doing
>>>>> wrong? One more thing - not sure if it is relevant : I am using the new
>>>>> Hadoop MapReduce API.
>>>>>
>>>>> Thanks in advance!
>>>>>
>>>>> Regards,
>>>>> Krishnakumar.
>>>
>>>
>>
Re: libjars options
Posted by "C.V.Krishnakumar Iyer" <f2...@gmail.com>.
Hi,
Thanks a lot! I shall try this once and let you know!
Regards,
Krishna.
On Jan 11, 2011, at 12:48 PM, Alex Kozlov wrote:
> There is also a blog that I recently wrote, if it helps
> http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job
>
> On Tue, Jan 11, 2011 at 12:33 PM, Alex Kozlov <al...@cloudera.com> wrote:
>
>> Have you implemented GenericOptionsParser? Do you see your jar in the *
>> mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view
>> via a JT Web UI)?
>>
>> --
>> Alex Kozlov
>> Solutions Architect
>> Cloudera, Inc
>> twitter: alexvk2009
>> <http://www.cloudera.com/company/press-center/hadoop-world-nyc/>
>>
>>
>> On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer <
>> f2004403@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have tried that as well, using -files <jar file> But it still gives the
>>> exact same error. Any other thing that I could try?
>>>
>>> Thanks,
>>> Krishna.
>>>
>>> On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:
>>>
>>>> Refer to Alex Kozlov's answer on 12/11/10
>>>>
>>>> On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
>>>> <f2...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Could anyone please guide me as to how to use the -libjars option in
>>> HDFS?
>>>>>
>>>>> I have added the necessary jar file (the hbase jar - to be precise) to
>>> the
>>>>> classpath of the node where I am starting the job.
>>>>>
>>>>> The following is the format that i am invoking:
>>>>> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars
>>> (separated by
>>>>> Commas)> <Arguments to our main class>
>>>>>
>>>>> bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars
>>>>> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a
>>>>>
>>>>> Despite this, I find that I get the java.lang.ClassNotFoundException
>>>>> error! :(
>>>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>>>> java.lang.ClassNotFoundException:
>>>>> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>>>>> at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
>>>>> at
>>>>>
>>> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
>>>>> at
>>>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
>>>>> at
>>>>> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
>>>>> at
>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>>> Caused by: java.lang.RuntimeException:
>>> java.lang.ClassNotFoundException:
>>>>> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>>>>> at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
>>>>> at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
>>>>>
>>>>> The strange thing is that there is another MR job I have that runs
>>>>> perfectly with the libjars option! Could anybody tell me what I am
>>> doing
>>>>> wrong? One more thing - not sure if it is relevant : I am using the new
>>>>> Hadoop MapReduce API.
>>>>>
>>>>> Thanks in advance!
>>>>>
>>>>> Regards,
>>>>> Krishnakumar.
>>>
>>>
>>
Re: libjars options
Posted by Alex Kozlov <al...@cloudera.com>.
There is also a blog that I recently wrote, if it helps
http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job
On Tue, Jan 11, 2011 at 12:33 PM, Alex Kozlov <al...@cloudera.com> wrote:
> Have you implemented GenericOptionsParser? Do you see your jar in the *
> mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view
> via a JT Web UI)?
>
> --
> Alex Kozlov
> Solutions Architect
> Cloudera, Inc
> twitter: alexvk2009
> <http://www.cloudera.com/company/press-center/hadoop-world-nyc/>
>
>
> On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer <
> f2004403@gmail.com> wrote:
>
>> Hi,
>>
>> I have tried that as well, using -files <jar file> But it still gives the
>> exact same error. Any other thing that I could try?
>>
>> Thanks,
>> Krishna.
>>
>> On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:
>>
>> > Refer to Alex Kozlov's answer on 12/11/10
>> >
>> > On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
>> > <f2...@gmail.com>wrote:
>> >
>> >> Hi,
>> >>
>> >> Could anyone please guide me as to how to use the -libjars option in
>> HDFS?
>> >>
>> >> I have added the necessary jar file (the hbase jar - to be precise) to
>> the
>> >> classpath of the node where I am starting the job.
>> >>
>> >> The following is the format that i am invoking:
>> >> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars
>> (separated by
>> >> Commas)> <Arguments to our main class>
>> >>
>> >> bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars
>> >> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a
>> >>
>> >> Despite this, I find that I get the java.lang.ClassNotFoundException
>> >> error! :(
>> >> java.lang.RuntimeException: java.lang.RuntimeException:
>> >> java.lang.ClassNotFoundException:
>> >> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>> >> at
>> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
>> >> at
>> >>
>> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
>> >> at
>> >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
>> >> at
>> >> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
>> >> at
>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> >> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >> Caused by: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException:
>> >> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>> >> at
>> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
>> >> at
>> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
>> >>
>> >> The strange thing is that there is another MR job I have that runs
>> >> perfectly with the libjars option! Could anybody tell me what I am
>> doing
>> >> wrong? One more thing - not sure if it is relevant : I am using the new
>> >> Hadoop MapReduce API.
>> >>
>> >> Thanks in advance!
>> >>
>> >> Regards,
>> >> Krishnakumar.
>>
>>
>
Re: libjars options
Posted by Alex Kozlov <al...@cloudera.com>.
Have you implemented GenericOptionsParser? Do you see your jar in the *
mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view
via a JT Web UI)?
--
Alex Kozlov
Solutions Architect
Cloudera, Inc
twitter: alexvk2009
<http://www.cloudera.com/company/press-center/hadoop-world-nyc/>
On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer
<f2...@gmail.com>wrote:
> Hi,
>
> I have tried that as well, using -files <jar file> But it still gives the
> exact same error. Any other thing that I could try?
>
> Thanks,
> Krishna.
>
> On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:
>
> > Refer to Alex Kozlov's answer on 12/11/10
> >
> > On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
> > <f2...@gmail.com>wrote:
> >
> >> Hi,
> >>
> >> Could anyone please guide me as to how to use the -libjars option in
> HDFS?
> >>
> >> I have added the necessary jar file (the hbase jar - to be precise) to
> the
> >> classpath of the node where I am starting the job.
> >>
> >> The following is the format that i am invoking:
> >> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated
> by
> >> Commas)> <Arguments to our main class>
> >>
> >> bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars
> >> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a
> >>
> >> Despite this, I find that I get the java.lang.ClassNotFoundException
> >> error! :(
> >> java.lang.RuntimeException: java.lang.RuntimeException:
> >> java.lang.ClassNotFoundException:
> >> org.apache.hadoop.hbase.io.ImmutableBytesWritable
> >> at
> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
> >> at
> >>
> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
> >> at
> >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
> >> at
> >> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
> >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> >> org.apache.hadoop.hbase.io.ImmutableBytesWritable
> >> at
> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> >> at
> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
> >>
> >> The strange thing is that there is another MR job I have that runs
> >> perfectly with the libjars option! Could anybody tell me what I am doing
> >> wrong? One more thing - not sure if it is relevant : I am using the new
> >> Hadoop MapReduce API.
> >>
> >> Thanks in advance!
> >>
> >> Regards,
> >> Krishnakumar.
>
>
Re: libjars options
Posted by "C.V.Krishnakumar Iyer" <f2...@gmail.com>.
Hi,
I have tried that as well, using -files <jar file> But it still gives the exact same error. Any other thing that I could try?
Thanks,
Krishna.
On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:
> Refer to Alex Kozlov's answer on 12/11/10
>
> On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
> <f2...@gmail.com>wrote:
>
>> Hi,
>>
>> Could anyone please guide me as to how to use the -libjars option in HDFS?
>>
>> I have added the necessary jar file (the hbase jar - to be precise) to the
>> classpath of the node where I am starting the job.
>>
>> The following is the format that i am invoking:
>> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated by
>> Commas)> <Arguments to our main class>
>>
>> bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars
>> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a
>>
>> Despite this, I find that I get the java.lang.ClassNotFoundException
>> error! :(
>> java.lang.RuntimeException: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException:
>> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
>> at
>> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
>> at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
>> at
>> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
>>
>> The strange thing is that there is another MR job I have that runs
>> perfectly with the libjars option! Could anybody tell me what I am doing
>> wrong? One more thing - not sure if it is relevant : I am using the new
>> Hadoop MapReduce API.
>>
>> Thanks in advance!
>>
>> Regards,
>> Krishnakumar.
Re: libjars options
Posted by Ted Yu <yu...@gmail.com>.
Refer to Alex Kozlov's answer on 12/11/10
On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
<f2...@gmail.com>wrote:
> Hi,
>
> Could anyone please guide me as to how to use the -libjars option in HDFS?
>
> I have added the necessary jar file (the hbase jar - to be precise) to the
> classpath of the node where I am starting the job.
>
> The following is the format that i am invoking:
> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated by
> Commas)> <Arguments to our main class>
>
> bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars
> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a
>
> Despite this, I find that I get the java.lang.ClassNotFoundException
> error! :(
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.io.ImmutableBytesWritable
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
> at
> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.io.ImmutableBytesWritable
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
>
> The strange thing is that there is another MR job I have that runs
> perfectly with the libjars option! Could anybody tell me what I am doing
> wrong? One more thing - not sure if it is relevant : I am using the new
> Hadoop MapReduce API.
>
> Thanks in advance!
>
> Regards,
> Krishnakumar.