You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Patrick Wendell (JIRA)" <ji...@apache.org> on 2014/03/30 06:09:14 UTC
[jira] [Updated] (SPARK-939) Allow user jars to take precedence
over Spark jars, if desired
[ https://issues.apache.org/jira/browse/SPARK-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated SPARK-939:
----------------------------------
Reporter: Patrick Wendell (was: Patrick Cogan)
> Allow user jars to take precedence over Spark jars, if desired
> --------------------------------------------------------------
>
> Key: SPARK-939
> URL: https://issues.apache.org/jira/browse/SPARK-939
> Project: Apache Spark
> Issue Type: New Feature
> Components: Spark Core
> Reporter: Patrick Wendell
> Assignee: holdenk
> Priority: Blocker
> Labels: starter
> Fix For: 1.0.0
>
>
> Sometimes a user may want to include their own version of a jar that spark itself uses. For example, if their code requires a newer version of that jar than Spark offers. It would be good to have an option to give the users dependencies precedence over Spark. This options should be disabled by default, since it could lead to some odd behavior (e.g. parts of Spark not working). But I think we should have it.
> From an implementation perspective, this would require modifying the way we do class loading inside of an Executor. The default behavior of the URLClassLoader is to delegate to it's parent first and, if that fails, to find a class locally. We want to have the opposite behavior. This is sometimes referred to as "parent-last" (as opposed to "parent-first") class loading precedence. There is an example of how to do this here:
> http://stackoverflow.com/questions/5445511/how-do-i-create-a-parent-last-child-first-classloader-in-java-or-how-to-overr
> We should write a similar class which can encapsulate a URL classloader and change the delegation order. Or if possible, maybe we could find a more elegant way to do this. See relevant discussion on the user list here:
> https://groups.google.com/forum/#!topic/spark-users/b278DW3e38g
> Also see the corresponding option in Hadoop:
> https://issues.apache.org/jira/browse/MAPREDUCE-4521
> Some other relevant Hadoop JIRA's:
> https://issues.apache.org/jira/browse/MAPREDUCE-1700
> https://issues.apache.org/jira/browse/MAPREDUCE-1938
--
This message was sent by Atlassian JIRA
(v6.2#6252)