[jira] [Commented] (SPARK-4877) userClassPathFirst doesn't handle user classes inheriting from parent

["Stephen Haberman (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14250461#comment-14250461 ] 

Stephen Haberman commented on SPARK-4877:
-----------------------------------------

Stack trace:

{code}
2014-12-17 05:07:36,811 WARN  [task-result-getter-0] scheduler.TaskSetManager (Logging.scala:logWarning(71)) - Lost task 0.0 in stage 0.0 (TID 0, ip-10-153-171-58.ec2.internal): java.lang.NoClassDefFoundError: org/apache/spark/serializer/KryoRegistrator
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at org.apache.spark.executor.ChildExecutorURLClassLoader$userClassLoader$.findClass(ExecutorURLClassLoader.scala:42)
	at org.apache.spark.executor.ChildExecutorURLClassLoader.findClass(ExecutorURLClassLoader.scala:50)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:270)
	at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$3.apply(KryoSerializer.scala:101)
	at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$3.apply(KryoSerializer.scala:101)
	at scala.Option.map(Option.scala:145)
	at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:101)
	at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:157)
	at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:119)
	at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:214)
	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:177)
	at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1000)
	at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:164)
	at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
	at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
	at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:87)
	at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:61)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
	at org.apache.spark.scheduler.Task.run(Task.scala:56)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.serializer.KryoRegistrator
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at org.apache.spark.executor.ChildExecutorURLClassLoader$userClassLoader$.findClass(ExecutorURLClassLoader.scala:42)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
{code}

Also note that I have a fix and will be submitting a PR soon.

> userClassPathFirst doesn't handle user classes inheriting from parent
> ---------------------------------------------------------------------
>
>                 Key: SPARK-4877
>                 URL: https://issues.apache.org/jira/browse/SPARK-4877
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.2.0
>            Reporter: Stephen Haberman
>
> We're trying out userClassPathFirst.
> To do so, we make an uberjar that does not contain Spark or Scala classes (because we want those to load from the parent classloader, otherwise we'll get errors like scala.Function0 != scala.Function0 since they'd load from different class loaders).
> (Tangentially, some isolation classloaders like Jetty whitelist certain packages, like spark/* and scala/*, to only come from the parent classloader, so that technically if the user still messes up and leaks the Scala/Spark jars into their uberjar, it won't blow up; this would be a good enhancement, I think.)
> Anyway, we have a custom Kryo registrar, which ships in our uberjar, but since it "extends spark.KryoRegistrator", which is not in our uberjar, we get a ClassNotFoundException.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org