You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@toree.apache.org by "Josh trier (JIRA)" <ji...@apache.org> on 2017/04/12 18:36:41 UTC
[jira] [Commented] (TOREE-355)
java.lang.IncompatibleClassChangeError: class
org.clapper.classutil.asm.ASMEmptyVisitor has interface
org.objectweb.asm.ClassVisitor as super class
[ https://issues.apache.org/jira/browse/TOREE-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966372#comment-15966372 ]
Josh trier commented on TOREE-355:
----------------------------------
I'm still hitting this problem. I pulled the toree master branch & built then installed the pip package in dist/ folder.
I'm connecting to an yarn cluster in the pyspark kernel.
Using cloudbreak/ambari which is deploying:
Amazon Linux AMI release 2016.09
HDP 2.6
openJDK 1.8
{quote}
17/04/12 18:22:13 INFO EmptyLocalActorRef: Message [org.apache.toree.kernel.protocol.v5.KernelMessage] from Actor[akka://spark-kernel-actor-system/user/kernel_message_relay#-1377949836] to Actor[akka://spark-kernel-actor-system/user/status] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
17/04/12 18:22:14 WARN Main$$anon$1: No external magics provided to PluginManager!
17/04/12 18:22:14 ERROR ApplicationMaster: User class threw exception: java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class
java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:250)
at org.clapper.classutil.ClassFinder.org$clapper$classutil$ClassFinder$$classData(ClassFinder.scala:427)
at org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
at org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
at scala.collection.immutable.Stream.map(Stream.scala:418)
at org.clapper.classutil.ClassFinder.processOpenZip(ClassFinder.scala:385)
at org.clapper.classutil.ClassFinder.processJar(ClassFinder.scala:340)
at org.clapper.classutil.ClassFinder.findClassesIn(ClassFinder.scala:329)
at org.clapper.classutil.ClassFinder.find(ClassFinder.scala:320)
at org.clapper.classutil.ClassFinder.getClasses(ClassFinder.scala:311)
at org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
at org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:73)
at org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:35)
at org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:34)
at org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:38)
at org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:38)
at org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45)
at org.apache.toree.plugins.PluginManager.internalPlugins(PluginManager.scala:44)
at org.apache.toree.plugins.PluginManager.initialize(PluginManager.scala:80)
at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializePlugins(ComponentInitialization.scala:221)
at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeComponents(ComponentInitialization.scala:86)
at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35)
at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:101)
at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:40)
at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at org.apache.toree.Main$.main(Main.scala:24)
at org.apache.toree.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
17/04/12 18:22:14 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class)
17/04/12 18:22:14 ERROR ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:194)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:401)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:254)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:766)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:764)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.util.concurrent.ExecutionException: Boxed Error
at scala.concurrent.impl.Promise$.resolver(Promise.scala:55)
at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244)
at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:655)
Caused by: java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:250)
at org.clapper.classutil.ClassFinder.org$clapper$classutil$ClassFinder$$classData(ClassFinder.scala:427)
at org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
at org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
at scala.collection.immutable.Stream.map(Stream.scala:418)
at org.clapper.classutil.ClassFinder.processOpenZip(ClassFinder.scala:385)
at org.clapper.classutil.ClassFinder.processJar(ClassFinder.scala:340)
at org.clapper.classutil.ClassFinder.findClassesIn(ClassFinder.scala:329)
at org.clapper.classutil.ClassFinder.find(ClassFinder.scala:320)
at org.clapper.classutil.ClassFinder.getClasses(ClassFinder.scala:311)
at org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
at org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:73)
at org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:35)
at org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:34)
at org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:38)
at org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:38)
at org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45)
at org.apache.toree.plugins.PluginManager.internalPlugins(PluginManager.scala:44)
at org.apache.toree.plugins.PluginManager.initialize(PluginManager.scala:80)
at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializePlugins(ComponentInitialization.scala:221)
at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeComponents(ComponentInitialization.scala:86)
at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35)
at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:101)
at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:40)
at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at org.apache.toree.Main$.main(Main.scala:24)
at org.apache.toree.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
17/04/12 18:22:14 INFO Main$$anon$1: Shutting down kernel
17/04/12 18:22:14 INFO Main$$anon$1: Shutting down interpreters
17/04/12 18:22:14 INFO Main$$anon$1: Shutting down actor system
17/04/12 18:22:14 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class)
17/04/12 18:22:14 INFO ApplicationMaster: Deleting staging directory hdfs://ip-10-0-153-71.ec2.internal:8020/user/testuser2/.sparkStaging/application_1492016613726_0213
17/04/12 18:22:14 INFO ShutdownHookManager: Shutdown hook called
{quote}
> java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: TOREE-355
> URL: https://issues.apache.org/jira/browse/TOREE-355
> Project: TOREE
> Issue Type: Bug
> Environment: Toree 0.2.0.dev1
> CDH 5.9's embedded spark-2.0.0-beta2
> Reporter: Adrien Lavoillotte
> Assignee: Luciano Resende
> Labels: easyfix
> Fix For: 0.2.0
>
>
> Using Toree 0.2.0.dev1 with CDH's embedded spark 2 did not cause TOREE-327 for me. Instead, I got this error:
> {code}
> Exception in thread "main" java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:250)
> at org.clapper.classutil.ClassFinder.org$clapper$classutil$ClassFinder$$classData(ClassFinder.scala:427)
> at org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
> at org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
> at scala.collection.immutable.Stream.map(Stream.scala:418)
> at org.clapper.classutil.ClassFinder.processOpenZip(ClassFinder.scala:385)
> at org.clapper.classutil.ClassFinder.processJar(ClassFinder.scala:340)
> at org.clapper.classutil.ClassFinder.findClassesIn(ClassFinder.scala:329)
> at org.clapper.classutil.ClassFinder.find(ClassFinder.scala:320)
> at org.clapper.classutil.ClassFinder.getClasses(ClassFinder.scala:311)
> at org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
> at org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
> at scala.util.Try$.apply(Try.scala:192)
> at org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:73)
> at org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:35)
> at org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:34)
> at org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:38)
> at org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:38)
> at org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45)
> at org.apache.toree.plugins.PluginManager.internalPlugins(PluginManager.scala:44)
> at org.apache.toree.plugins.PluginManager.initialize(PluginManager.scala:80)
> at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializePlugins(ComponentInitialization.scala:221)
> at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeComponents(ComponentInitialization.scala:86)
> at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35)
> at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:101)
> at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:40)
> at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24)
> {code}
> Apparently in some much older version of ASM (3), {{ClassVisitor}} was an interface and not an abstract class. Said much older version can be found in some Hadoop libs or older Spark libs, and ends up in the classpath with CDH's spark 2.
> The simplest solution is to shadow ASM, e.g. using JarJAr Links. The following produces an assembly jar that does not have this issue:
> {code}
> echo 'rule org.objectweb.asm.** org.apache.toree.shadow.@0' >relocate.rules
> java -jar jarjar-1.4.jar process relocate.rules toree.jar toree-relocated.jar
> {code}
> ASM is a dependency of ClassUtil, for which [an issue on this is currently open|https://github.com/bmc/classutil/issues/19], but the author says he may not want to shadow ASM, so I thought maybe you might want to.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)