You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Barak Yaish <ba...@gmail.com> on 2013/03/07 19:48:14 UTC
Job driver and 3rd party jars
Hi,
I'm able to run M/R jobs where the mapper and reducer required to use 3rd
party jars. I'm registering those jars in -libjar while invoking the hadoop
jar command. I'm facing a strange problem, though, when the job driver
itself ( extends Configured implements Tool ) required to run such code (
for example notify some remote service upon start and end). Is there a way
to configure classpath when submitting jobs using hadoop jar? Seems like
-libjar doesn't work for this case...
Exception in thread "main" java.lang.NoClassDefFoundError:
com/me/context/DefaultContext
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at
com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: com.me.context.DefaultContext
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Re: Job driver and 3rd party jars
Posted by Barak Yaish <ba...@gmail.com>.
Yep, my typo, I'm using the later. I was also trying export
HADOOP_CLASSPATH_USER_FIRST =true and export HADOOP_CLASSPATH=myjar before
launching the hadoop jar, but I still getting the same exception.
I'm running hadoop 1.0.4.
On Mar 8, 2013 2:27 AM, "Harsh J" <ha...@cloudera.com> wrote:
> To be precise, did you use -libjar or -libjars? The latter is the right
> option.
>
> On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <ba...@gmail.com>
> wrote:
> > Hi,
> >
> > I'm able to run M/R jobs where the mapper and reducer required to use 3rd
> > party jars. I'm registering those jars in -libjar while invoking the
> hadoop
> > jar command. I'm facing a strange problem, though, when the job driver
> > itself ( extends Configured implements Tool ) required to run such code (
> > for example notify some remote service upon start and end). Is there a
> way
> > to configure classpath when submitting jobs using hadoop jar? Seems like
> > -libjar doesn't work for this case...
> >
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > com/me/context/DefaultContext
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > at
> > com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> > at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > Caused by: java.lang.ClassNotFoundException:
> com.me.context.DefaultContext
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>
>
>
> --
> Harsh J
>
Re: Job driver and 3rd party jars
Posted by Barak Yaish <ba...@gmail.com>.
Yep, my typo, I'm using the later. I was also trying export
HADOOP_CLASSPATH_USER_FIRST =true and export HADOOP_CLASSPATH=myjar before
launching the hadoop jar, but I still getting the same exception.
I'm running hadoop 1.0.4.
On Mar 8, 2013 2:27 AM, "Harsh J" <ha...@cloudera.com> wrote:
> To be precise, did you use -libjar or -libjars? The latter is the right
> option.
>
> On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <ba...@gmail.com>
> wrote:
> > Hi,
> >
> > I'm able to run M/R jobs where the mapper and reducer required to use 3rd
> > party jars. I'm registering those jars in -libjar while invoking the
> hadoop
> > jar command. I'm facing a strange problem, though, when the job driver
> > itself ( extends Configured implements Tool ) required to run such code (
> > for example notify some remote service upon start and end). Is there a
> way
> > to configure classpath when submitting jobs using hadoop jar? Seems like
> > -libjar doesn't work for this case...
> >
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > com/me/context/DefaultContext
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > at
> > com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> > at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > Caused by: java.lang.ClassNotFoundException:
> com.me.context.DefaultContext
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>
>
>
> --
> Harsh J
>
Re: Job driver and 3rd party jars
Posted by Barak Yaish <ba...@gmail.com>.
Yep, my typo, I'm using the later. I was also trying export
HADOOP_CLASSPATH_USER_FIRST =true and export HADOOP_CLASSPATH=myjar before
launching the hadoop jar, but I still getting the same exception.
I'm running hadoop 1.0.4.
On Mar 8, 2013 2:27 AM, "Harsh J" <ha...@cloudera.com> wrote:
> To be precise, did you use -libjar or -libjars? The latter is the right
> option.
>
> On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <ba...@gmail.com>
> wrote:
> > Hi,
> >
> > I'm able to run M/R jobs where the mapper and reducer required to use 3rd
> > party jars. I'm registering those jars in -libjar while invoking the
> hadoop
> > jar command. I'm facing a strange problem, though, when the job driver
> > itself ( extends Configured implements Tool ) required to run such code (
> > for example notify some remote service upon start and end). Is there a
> way
> > to configure classpath when submitting jobs using hadoop jar? Seems like
> > -libjar doesn't work for this case...
> >
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > com/me/context/DefaultContext
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > at
> > com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> > at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > Caused by: java.lang.ClassNotFoundException:
> com.me.context.DefaultContext
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>
>
>
> --
> Harsh J
>
Re: Job driver and 3rd party jars
Posted by Barak Yaish <ba...@gmail.com>.
Yep, my typo, I'm using the later. I was also trying export
HADOOP_CLASSPATH_USER_FIRST =true and export HADOOP_CLASSPATH=myjar before
launching the hadoop jar, but I still getting the same exception.
I'm running hadoop 1.0.4.
On Mar 8, 2013 2:27 AM, "Harsh J" <ha...@cloudera.com> wrote:
> To be precise, did you use -libjar or -libjars? The latter is the right
> option.
>
> On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <ba...@gmail.com>
> wrote:
> > Hi,
> >
> > I'm able to run M/R jobs where the mapper and reducer required to use 3rd
> > party jars. I'm registering those jars in -libjar while invoking the
> hadoop
> > jar command. I'm facing a strange problem, though, when the job driver
> > itself ( extends Configured implements Tool ) required to run such code (
> > for example notify some remote service upon start and end). Is there a
> way
> > to configure classpath when submitting jobs using hadoop jar? Seems like
> > -libjar doesn't work for this case...
> >
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > com/me/context/DefaultContext
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> > at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > at
> > com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> > at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > Caused by: java.lang.ClassNotFoundException:
> com.me.context.DefaultContext
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>
>
>
> --
> Harsh J
>
Re: Job driver and 3rd party jars
Posted by Harsh J <ha...@cloudera.com>.
To be precise, did you use -libjar or -libjars? The latter is the right option.
On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <ba...@gmail.com> wrote:
> Hi,
>
> I'm able to run M/R jobs where the mapper and reducer required to use 3rd
> party jars. I'm registering those jars in -libjar while invoking the hadoop
> jar command. I'm facing a strange problem, though, when the job driver
> itself ( extends Configured implements Tool ) required to run such code (
> for example notify some remote service upon start and end). Is there a way
> to configure classpath when submitting jobs using hadoop jar? Seems like
> -libjar doesn't work for this case...
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/me/context/DefaultContext
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at
> com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException: com.me.context.DefaultContext
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
--
Harsh J
Re: Job driver and 3rd party jars
Posted by Harsh J <ha...@cloudera.com>.
To be precise, did you use -libjar or -libjars? The latter is the right option.
On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <ba...@gmail.com> wrote:
> Hi,
>
> I'm able to run M/R jobs where the mapper and reducer required to use 3rd
> party jars. I'm registering those jars in -libjar while invoking the hadoop
> jar command. I'm facing a strange problem, though, when the job driver
> itself ( extends Configured implements Tool ) required to run such code (
> for example notify some remote service upon start and end). Is there a way
> to configure classpath when submitting jobs using hadoop jar? Seems like
> -libjar doesn't work for this case...
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/me/context/DefaultContext
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at
> com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException: com.me.context.DefaultContext
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
--
Harsh J
Re: Job driver and 3rd party jars
Posted by Harsh J <ha...@cloudera.com>.
To be precise, did you use -libjar or -libjars? The latter is the right option.
On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <ba...@gmail.com> wrote:
> Hi,
>
> I'm able to run M/R jobs where the mapper and reducer required to use 3rd
> party jars. I'm registering those jars in -libjar while invoking the hadoop
> jar command. I'm facing a strange problem, though, when the job driver
> itself ( extends Configured implements Tool ) required to run such code (
> for example notify some remote service upon start and end). Is there a way
> to configure classpath when submitting jobs using hadoop jar? Seems like
> -libjar doesn't work for this case...
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/me/context/DefaultContext
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at
> com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException: com.me.context.DefaultContext
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
--
Harsh J
Re: Job driver and 3rd party jars
Posted by Harsh J <ha...@cloudera.com>.
To be precise, did you use -libjar or -libjars? The latter is the right option.
On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <ba...@gmail.com> wrote:
> Hi,
>
> I'm able to run M/R jobs where the mapper and reducer required to use 3rd
> party jars. I'm registering those jars in -libjar while invoking the hadoop
> jar command. I'm facing a strange problem, though, when the job driver
> itself ( extends Configured implements Tool ) required to run such code (
> for example notify some remote service upon start and end). Is there a way
> to configure classpath when submitting jobs using hadoop jar? Seems like
> -libjar doesn't work for this case...
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/me/context/DefaultContext
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at
> com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException: com.me.context.DefaultContext
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
--
Harsh J