[jira] [Commented] (SPARK-1863) Allowing user jars to take precedence over Spark jars does not work as expected

["koert kuipers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000512#comment-14000512 ] 

koert kuipers commented on SPARK-1863:
--------------------------------------

When i addresses this by creating a new version of ChildExecutorURLClassLoader that does have the proper parent-child relationship and reverses the loading order inside loadClass the class loading seemed to work fine but now classes like SparkEnv were loaded by ChildExecutorURLClassLoader leading to NPEs on SparkEnv.get()

To verify that the issue was that SparkEnv was now loaded by ChildExecutorURLClassLoader i forced SparkEnv to be loaded by the parent classloader. That didnt help. Then i forced all spark classes to be loaded by parent classloader and that did help. But it causes even bigger problems:

java.lang.LinkageError: loader constraint violation: when resolving overridden method "myclass.MyRDD.compute(Lorg/apache/spark/Partition;Lorg/apache/spark/TaskContext;)Lscala/collection/Iterator;" the class loader (instance of org/apache/spark/executor/ChildExecutorURLClassLoader) of the current class, myclass/MyRDD, and its superclass loader (instance of sun/misc/Launcher$AppClassLoader), have different Class objects for the type TaskContext;)Lscala/collection/Iterator; used in the signature


> Allowing user jars to take precedence over Spark jars does not work as expected
> -------------------------------------------------------------------------------
>
>                 Key: SPARK-1863
>                 URL: https://issues.apache.org/jira/browse/SPARK-1863
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.0.0
>            Reporter: koert kuipers
>            Priority: Minor
>
> See here:
> http://apache-spark-user-list.1001560.n3.nabble.com/java-serialization-errors-with-spark-files-userClassPathFirst-true-td5832.html
> The issue seems to be that within ChildExecutorURLClassLoader userClassLoader has no visibility on classes managed by parentClassLoader because their is no parent/child relationship. What this means that if a class is loaded by userClassLoader and it refers to a class loaded by parentClassLoader you get a NoClassDefFoundError.



--
This message was sent by Atlassian JIRA
(v6.2#6252)