You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by pr...@nokia.com on 2010/12/29 16:40:03 UTC
External jar dependency of map reduce jobs
Hello all,
I have few map reduce jobs that I am invoking from a glasfish container. I am not using "hadoop" command line tool but calling the jobs directly from glassfish programatically. I have a driver that runs on glassfish and calls these jobs sequentially. I was able to run the jobs as long as they do not depend on external jar files. But I have one job that depends on external jars and that fails with ClassNotFound exception.
I know that -libjars options can be used when using command line hadoop but since I am invoking the jobs programatically from an app server, I am wondering how I can add external dependencies in this case.
Any help would be appreciated.
Thanks
Praveen
RE: External jar dependency of map reduce jobs
Posted by "Halai, Murtaza" <mu...@amazon.com>.
Hi,
In out build.xml we have a target that creates a standalone.jar that contains the class files belonging to the project and also includes the jars for all transitive dependencies. We use this jar to launch our jobs and in the jobConf if you set your mapred.jar to this jar things should work.
See: https://p4db.amazon.com/source-views/p4db/brazil/fileViewer.cgi?FSPC=//brazil/src/appgroup/duplicateDetection/apps/DarwinInferredVariationsService/mainline/build.xml&REV=10
Murtaza
From: Hiller, Dean (Contractor) [mailto:dean.hiller@broadridge.com]
Sent: Wednesday, December 29, 2010 3:03 PM
To: mapreduce-user@hadoop.apache.org
Subject: RE: External jar dependency of map reduce jobs
Google DistributedCache hadoop...I think there is a method for DistributedCache.addResource though not sure if adding just text files or jars, but maybe that will help you in looking. Sorry, I am a newb as well.
Dean
From: praveen.peddi@nokia.com [mailto:praveen.peddi@nokia.com]
Sent: Wednesday, December 29, 2010 8:40 AM
To: mapreduce-user@hadoop.apache.org
Subject: External jar dependency of map reduce jobs
Hello all,
I have few map reduce jobs that I am invoking from a glasfish container. I am not using "hadoop" command line tool but calling the jobs directly from glassfish programatically. I have a driver that runs on glassfish and calls these jobs sequentially. I was able to run the jobs as long as they do not depend on external jar files. But I have one job that depends on external jars and that fails with ClassNotFound exception.
I know that -libjars options can be used when using command line hadoop but since I am invoking the jobs programatically from an app server, I am wondering how I can add external dependencies in this case.
Any help would be appreciated.
Thanks
Praveen
This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
RE: External jar dependency of map reduce jobs
Posted by "Hiller, Dean (Contractor)" <de...@broadridge.com>.
Google DistributedCache hadoop...I think there is a method for
DistributedCache.addResource though not sure if adding just text files
or jars, but maybe that will help you in looking. Sorry, I am a newb as
well.
Dean
From: praveen.peddi@nokia.com [mailto:praveen.peddi@nokia.com]
Sent: Wednesday, December 29, 2010 8:40 AM
To: mapreduce-user@hadoop.apache.org
Subject: External jar dependency of map reduce jobs
Hello all,
I have few map reduce jobs that I am invoking from a glasfish container.
I am not using "hadoop" command line tool but calling the jobs directly
from glassfish programatically. I have a driver that runs on glassfish
and calls these jobs sequentially. I was able to run the jobs as long as
they do not depend on external jar files. But I have one job that
depends on external jars and that fails with ClassNotFound exception.
I know that -libjars options can be used when using command line hadoop
but since I am invoking the jobs programatically from an app server, I
am wondering how I can add external dependencies in this case.
Any help would be appreciated.
Thanks
Praveen
This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
Re: External jar dependency of map reduce jobs
Posted by Harsh J <qw...@gmail.com>.
Add them to the distributed cache, as must be done with any local file
that needs distribution to Task JVMs.
DistributedCache.addFileToClassPath(...) should help.
--
Harsh J
www.harshj.com