You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Zheyi RONG <ro...@gmail.com> on 2013/03/14 14:12:17 UTC
Run jar built with new apis on a CDH3 cluster
Hi,
Is it possible to run a jar built with hadoop new apis (hadoop 0.22.0) on a
cluster deployed with CDH 3 (hadoop 0.20.2) ?
The jar uses some hadoop Api (e.g.
org.apache.hadoop.mapreduce.lib.chain.ChainMapper) that hadoop 0.20.2 does
not have.
I ran it, but hadoop complains:
Exception in thread "main" java.lang.IncompatibleClassChangeError:
Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at
org.apache.hadoop.mapreduce.lib.chain.ChainMapper.addMapper(ChainMapper.java:126)
at driver.ChainDriver.run(ChainDriver.java:146)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at driver.ChainDriver.main(ChainDriver.java:173)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Is there any workaround? I really do not want to rewrite my project...
Thank you very much.
Regards,
Zheyi.
Re: Run jar built with new apis on a CDH3 cluster
Posted by Harsh J <ha...@cloudera.com>.
It isn't possible to do this since 0.21+ broke MR binary compatibility
aside of some other minor API changes. You do not need to rewrite your
application/job in most cases, but you do have to recompile it against the
target cluster's libraries to get a binary compatible jar you can run.
On Thu, Mar 14, 2013 at 6:42 PM, Zheyi RONG <ro...@gmail.com> wrote:
> Hi,
>
> Is it possible to run a jar built with hadoop new apis (hadoop 0.22.0) on
> a cluster deployed with CDH 3 (hadoop 0.20.2) ?
> The jar uses some hadoop Api (e.g.
> org.apache.hadoop.mapreduce.lib.chain.ChainMapper) that hadoop 0.20.2 does
> not have.
>
> I ran it, but hadoop complains:
> Exception in thread "main" java.lang.IncompatibleClassChangeError:
> Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at
> org.apache.hadoop.mapreduce.lib.chain.ChainMapper.addMapper(ChainMapper.java:126)
> at driver.ChainDriver.run(ChainDriver.java:146)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at driver.ChainDriver.main(ChainDriver.java:173)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>
>
> Is there any workaround? I really do not want to rewrite my project...
>
> Thank you very much.
>
> Regards,
>
> Zheyi.
>
--
Harsh J
Re: Run jar built with new apis on a CDH3 cluster
Posted by Harsh J <ha...@cloudera.com>.
It isn't possible to do this since 0.21+ broke MR binary compatibility
aside of some other minor API changes. You do not need to rewrite your
application/job in most cases, but you do have to recompile it against the
target cluster's libraries to get a binary compatible jar you can run.
On Thu, Mar 14, 2013 at 6:42 PM, Zheyi RONG <ro...@gmail.com> wrote:
> Hi,
>
> Is it possible to run a jar built with hadoop new apis (hadoop 0.22.0) on
> a cluster deployed with CDH 3 (hadoop 0.20.2) ?
> The jar uses some hadoop Api (e.g.
> org.apache.hadoop.mapreduce.lib.chain.ChainMapper) that hadoop 0.20.2 does
> not have.
>
> I ran it, but hadoop complains:
> Exception in thread "main" java.lang.IncompatibleClassChangeError:
> Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at
> org.apache.hadoop.mapreduce.lib.chain.ChainMapper.addMapper(ChainMapper.java:126)
> at driver.ChainDriver.run(ChainDriver.java:146)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at driver.ChainDriver.main(ChainDriver.java:173)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>
>
> Is there any workaround? I really do not want to rewrite my project...
>
> Thank you very much.
>
> Regards,
>
> Zheyi.
>
--
Harsh J
Re: Run jar built with new apis on a CDH3 cluster
Posted by Harsh J <ha...@cloudera.com>.
It isn't possible to do this since 0.21+ broke MR binary compatibility
aside of some other minor API changes. You do not need to rewrite your
application/job in most cases, but you do have to recompile it against the
target cluster's libraries to get a binary compatible jar you can run.
On Thu, Mar 14, 2013 at 6:42 PM, Zheyi RONG <ro...@gmail.com> wrote:
> Hi,
>
> Is it possible to run a jar built with hadoop new apis (hadoop 0.22.0) on
> a cluster deployed with CDH 3 (hadoop 0.20.2) ?
> The jar uses some hadoop Api (e.g.
> org.apache.hadoop.mapreduce.lib.chain.ChainMapper) that hadoop 0.20.2 does
> not have.
>
> I ran it, but hadoop complains:
> Exception in thread "main" java.lang.IncompatibleClassChangeError:
> Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at
> org.apache.hadoop.mapreduce.lib.chain.ChainMapper.addMapper(ChainMapper.java:126)
> at driver.ChainDriver.run(ChainDriver.java:146)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at driver.ChainDriver.main(ChainDriver.java:173)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>
>
> Is there any workaround? I really do not want to rewrite my project...
>
> Thank you very much.
>
> Regards,
>
> Zheyi.
>
--
Harsh J
Re: Run jar built with new apis on a CDH3 cluster
Posted by Harsh J <ha...@cloudera.com>.
It isn't possible to do this since 0.21+ broke MR binary compatibility
aside of some other minor API changes. You do not need to rewrite your
application/job in most cases, but you do have to recompile it against the
target cluster's libraries to get a binary compatible jar you can run.
On Thu, Mar 14, 2013 at 6:42 PM, Zheyi RONG <ro...@gmail.com> wrote:
> Hi,
>
> Is it possible to run a jar built with hadoop new apis (hadoop 0.22.0) on
> a cluster deployed with CDH 3 (hadoop 0.20.2) ?
> The jar uses some hadoop Api (e.g.
> org.apache.hadoop.mapreduce.lib.chain.ChainMapper) that hadoop 0.20.2 does
> not have.
>
> I ran it, but hadoop complains:
> Exception in thread "main" java.lang.IncompatibleClassChangeError:
> Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at
> org.apache.hadoop.mapreduce.lib.chain.ChainMapper.addMapper(ChainMapper.java:126)
> at driver.ChainDriver.run(ChainDriver.java:146)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at driver.ChainDriver.main(ChainDriver.java:173)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>
>
> Is there any workaround? I really do not want to rewrite my project...
>
> Thank you very much.
>
> Regards,
>
> Zheyi.
>
--
Harsh J