You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mark <st...@gmail.com> on 2010/08/29 07:29:14 UTC
Classpath
How can I add jars to Hadoops classpath when running MapReduce jobs
for the following situations?
1) Assuming that the jars are local the nodes that running the job.
2) The jobs are only local to the client submitting the job.
I'm assuming I can just jar up all required jobs into the main job jar
being submitted, but I was wondering if there was some other way. Thanks
Re: Classpath
Posted by Alejandro Abdelnur <tu...@gmail.com>.
Yes, you can do #1, but I wouldn't say it is practical. You can do #2
as well, as you suggest.
But, IMO, the best way is copying the JARs in HDFS and using DistributedCache.
A
On Sun, Aug 29, 2010 at 1:29 PM, Mark <st...@gmail.com> wrote:
> How can I add jars to Hadoops classpath when running MapReduce jobs for the
> following situations?
>
> 1) Assuming that the jars are local the nodes that running the job.
> 2) The jobs are only local to the client submitting the job.
>
> I'm assuming I can just jar up all required jobs into the main job jar being
> submitted, but I was wondering if there was some other way. Thanks
>
Re: Classpath
Posted by Alex Baranau <al...@gmail.com>.
>From http://blog.sematext.com/2010/05/31/hadoop-digest-may-2010/ FAQ
section:
"How can I attach external libraries (jars) which my jobs depend on?
You can put them in a “lib” subdirectory of your jar root directory.
Alternatively you can use DistributedCache API."
Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - HBase
On Sun, Aug 29, 2010 at 8:29 AM, Mark <st...@gmail.com> wrote:
> How can I add jars to Hadoops classpath when running MapReduce jobs for
> the following situations?
>
> 1) Assuming that the jars are local the nodes that running the job.
> 2) The jobs are only local to the client submitting the job.
>
> I'm assuming I can just jar up all required jobs into the main job jar
> being submitted, but I was wondering if there was some other way. Thanks
>
Re: Classpath
Posted by Amareshwari Sri Ramadasu <am...@yahoo-inc.com>.
It works on the client side also if the files are on the local file system.
On 8/30/10 9:15 PM, "Mark" <st...@gmail.com> wrote:
On 8/30/10 7:38 AM, Mark wrote:
> On 8/29/10 10:38 PM, Amareshwari Sri Ramadasu wrote:
>> You can use -libjars option.
>>
>>
>> On 8/29/10 10:59 AM, "Mark"<st...@gmail.com> wrote:
>>
>> How can I add jars to Hadoops classpath when running MapReduce jobs
>> for the following situations?
>>
>> 1) Assuming that the jars are local the nodes that running the job.
>> 2) The jobs are only local to the client submitting the job.
>>
>> I'm assuming I can just jar up all required jobs into the main job jar
>> being submitted, but I was wondering if there was some other way. Thanks
>>
>>
> thanks. thats what I was looking for
This doesn't work though however from the client side does it?
Re: Classpath
Posted by Mark <st...@gmail.com>.
On 8/30/10 7:38 AM, Mark wrote:
> On 8/29/10 10:38 PM, Amareshwari Sri Ramadasu wrote:
>> You can use -libjars option.
>>
>>
>> On 8/29/10 10:59 AM, "Mark"<st...@gmail.com> wrote:
>>
>> How can I add jars to Hadoops classpath when running MapReduce jobs
>> for the following situations?
>>
>> 1) Assuming that the jars are local the nodes that running the job.
>> 2) The jobs are only local to the client submitting the job.
>>
>> I'm assuming I can just jar up all required jobs into the main job jar
>> being submitted, but I was wondering if there was some other way. Thanks
>>
>>
> thanks. thats what I was looking for
This doesn't work though however from the client side does it?
Re: Classpath
Posted by Mark <st...@gmail.com>.
On 8/29/10 10:38 PM, Amareshwari Sri Ramadasu wrote:
> You can use -libjars option.
>
>
> On 8/29/10 10:59 AM, "Mark"<st...@gmail.com> wrote:
>
> How can I add jars to Hadoops classpath when running MapReduce jobs
> for the following situations?
>
> 1) Assuming that the jars are local the nodes that running the job.
> 2) The jobs are only local to the client submitting the job.
>
> I'm assuming I can just jar up all required jobs into the main job jar
> being submitted, but I was wondering if there was some other way. Thanks
>
>
thanks. thats what I was looking for
Re: Classpath
Posted by Amareshwari Sri Ramadasu <am...@yahoo-inc.com>.
You can use -libjars option.
On 8/29/10 10:59 AM, "Mark" <st...@gmail.com> wrote:
How can I add jars to Hadoops classpath when running MapReduce jobs
for the following situations?
1) Assuming that the jars are local the nodes that running the job.
2) The jobs are only local to the client submitting the job.
I'm assuming I can just jar up all required jobs into the main job jar
being submitted, but I was wondering if there was some other way. Thanks