You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by pr...@nokia.com on 2010/12/29 16:40:03 UTC

External jar dependency of map reduce jobs

Hello all,
I have few map reduce jobs that I am invoking from a glasfish container. I am not using "hadoop" command line tool but calling the jobs directly from glassfish programatically. I have a driver that runs on glassfish and calls these jobs sequentially. I was able to run the jobs as long as they do not depend on external jar files. But I have one job that depends on external jars and that fails with ClassNotFound exception.

I know that -libjars options can be used when using command line hadoop but since I am invoking the jobs programatically from an app server, I am wondering how I can add external dependencies in this case.

Any help would be appreciated.

Thanks
Praveen


RE: External jar dependency of map reduce jobs

Posted by "Halai, Murtaza" <mu...@amazon.com>.
Hi,

In out build.xml we have a target that creates a standalone.jar that contains the class files belonging to the project and also includes the jars for all transitive dependencies. We use this jar to launch our jobs and in the jobConf if you set your mapred.jar to this jar  things should work.
See: https://p4db.amazon.com/source-views/p4db/brazil/fileViewer.cgi?FSPC=//brazil/src/appgroup/duplicateDetection/apps/DarwinInferredVariationsService/mainline/build.xml&REV=10

Murtaza



From: Hiller, Dean (Contractor) [mailto:dean.hiller@broadridge.com]
Sent: Wednesday, December 29, 2010 3:03 PM
To: mapreduce-user@hadoop.apache.org
Subject: RE: External jar dependency of map reduce jobs

Google DistributedCache hadoop...I think there is a method for DistributedCache.addResource though not sure if adding just text files or jars, but maybe that will help you in looking.  Sorry, I am a newb as well.
Dean

From: praveen.peddi@nokia.com [mailto:praveen.peddi@nokia.com]
Sent: Wednesday, December 29, 2010 8:40 AM
To: mapreduce-user@hadoop.apache.org
Subject: External jar dependency of map reduce jobs

Hello all,
I have few map reduce jobs that I am invoking from a glasfish container. I am not using "hadoop" command line tool but calling the jobs directly from glassfish programatically. I have a driver that runs on glassfish and calls these jobs sequentially. I was able to run the jobs as long as they do not depend on external jar files. But I have one job that depends on external jars and that fails with ClassNotFound exception.

I know that -libjars options can be used when using command line hadoop but since I am invoking the jobs programatically from an app server, I am wondering how I can add external dependencies in this case.

Any help would be appreciated.

Thanks
Praveen


This message and any attachments are intended only for the use of the addressee and

may contain information that is privileged and confidential. If the reader of the

message is not the intended recipient or an authorized representative of the

intended recipient, you are hereby notified that any dissemination of this

communication is strictly prohibited. If you have received this communication in

error, please notify us immediately by e-mail and delete the message and any

attachments from your system.



RE: External jar dependency of map reduce jobs

Posted by "Hiller, Dean (Contractor)" <de...@broadridge.com>.
Google DistributedCache hadoop...I think there is a method for
DistributedCache.addResource though not sure if adding just text files
or jars, but maybe that will help you in looking.  Sorry, I am a newb as
well.

Dean

 

From: praveen.peddi@nokia.com [mailto:praveen.peddi@nokia.com] 
Sent: Wednesday, December 29, 2010 8:40 AM
To: mapreduce-user@hadoop.apache.org
Subject: External jar dependency of map reduce jobs

 

Hello all,

I have few map reduce jobs that I am invoking from a glasfish container.
I am not using "hadoop" command line tool but calling the jobs directly
from glassfish programatically. I have a driver that runs on glassfish
and calls these jobs sequentially. I was able to run the jobs as long as
they do not depend on external jar files. But I have one job that
depends on external jars and that fails with ClassNotFound exception.

 

I know that -libjars options can be used when using command line hadoop
but since I am invoking the jobs programatically from an app server, I
am wondering how I can add external dependencies in this case.

 

Any help would be appreciated.

 

Thanks

Praveen

 


This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the 
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.


Re: External jar dependency of map reduce jobs

Posted by Harsh J <qw...@gmail.com>.
Add them to the distributed cache, as must be done with any local file
that needs distribution to Task JVMs.
DistributedCache.addFileToClassPath(...) should help.

-- 
Harsh J
www.harshj.com